λ

Lab Monitor

REAL-TIME EXPERIMENT DASHBOARD

LIVE
--:--:--

实验数据总览

active experiments: 8  ·  timezone: UTC+8  ·  sorted by: last update
#01

TRACEHCR-EBfinalv3

▦ table
last update
260822-00:48:04
records
2000
type
text
latest
tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:36008s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:6182s | GPU cuda1:7751/24564MiB:32%;cuda2:11435/24564MiB:22%
ID数值接收时间
3476tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:36008s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:6182s | GPU cuda1:7751/24564MiB:32%;cuda2:11435/24564MiB:22%260822-00:48:04 (UTC+8)
3475tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:35825s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:6000s | GPU cuda1:7751/24564MiB:17%;cuda2:11435/24564MiB:100%260822-00:45:03 (UTC+8)
3474tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:35643s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:5818s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260822-00:42:01 (UTC+8)
3473tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:35461s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:5636s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260822-00:38:58 (UTC+8)
3472tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:35279s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:5454s | GPU cuda1:7751/24564MiB:14%;cuda2:11435/24564MiB:100%260822-00:35:56 (UTC+8)
3471tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:35098s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:5273s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260822-00:32:54 (UTC+8)
3470tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:34915s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:5090s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260822-00:29:53 (UTC+8)
3469tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:34733s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:4908s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:35%260822-00:26:51 (UTC+8)
3468tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:34551s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:4726s | GPU cuda1:7751/24564MiB:26%;cuda2:11435/24564MiB:100%260822-00:23:48 (UTC+8)
3467tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:34369s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:4544s | GPU cuda1:7751/24564MiB:27%;cuda2:11435/24564MiB:47%260822-00:20:46 (UTC+8)
3466tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:34187s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:4362s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:74%260822-00:17:44 (UTC+8)
3465tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:34005s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:4180s | GPU cuda1:7751/24564MiB:33%;cuda2:11435/24564MiB:72%260822-00:14:42 (UTC+8)
3464tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:33822s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:3997s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:99%260822-00:11:40 (UTC+8)
3463tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:33640s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:3815s | GPU cuda1:7751/24564MiB:11%;cuda2:11435/24564MiB:100%260822-00:08:37 (UTC+8)
3462tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:33459s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:3634s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:83%260822-00:05:35 (UTC+8)
3461tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:33277s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:3452s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:100%260822-00:02:34 (UTC+8)
3460tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:33095s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:3270s | GPU cuda1:7751/24564MiB:16%;cuda2:11435/24564MiB:100%260821-23:59:32 (UTC+8)
3459tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:32912s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:3087s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:81%260821-23:56:30 (UTC+8)
3458tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:32729s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:2904s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:85%260821-23:53:27 (UTC+8)
3457tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:32545s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:2720s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-23:50:24 (UTC+8)
3456tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:32363s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:2538s | GPU cuda1:7751/24564MiB:16%;cuda2:11435/24564MiB:100%260821-23:47:20 (UTC+8)
3455tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:32182s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:2357s | GPU cuda1:7751/24564MiB:31%;cuda2:11435/24564MiB:100%260821-23:44:19 (UTC+8)
3454tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:32000s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:2175s | GPU cuda1:7751/24564MiB:37%;cuda2:11435/24564MiB:92%260821-23:41:17 (UTC+8)
3453tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:31818s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:1993s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260821-23:38:15 (UTC+8)
3452tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:31636s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:1811s | GPU cuda1:7751/24564MiB:16%;cuda2:11435/24564MiB:100%260821-23:35:13 (UTC+8)
3451tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:31454s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:1629s | GPU cuda1:7751/24564MiB:17%;cuda2:11435/24564MiB:49%260821-23:32:11 (UTC+8)
3450tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:31272s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:1446s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:100%260821-23:29:09 (UTC+8)
3449tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:31090s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:1265s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-23:26:07 (UTC+8)
3448tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:30908s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:1082s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-23:23:05 (UTC+8)
3447tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:30726s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:900s | GPU cuda1:7751/24564MiB:37%;cuda2:11435/24564MiB:10%260821-23:20:03 (UTC+8)
3446tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:30543s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:718s | GPU cuda1:7751/24564MiB:45%;cuda2:11435/24564MiB:100%260821-23:17:01 (UTC+8)
3445tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:30359s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:534s | GPU cuda1:7751/24564MiB:39%;cuda2:11435/24564MiB:63%260821-23:13:58 (UTC+8)
3444tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:30177s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:352s | GPU cuda1:7751/24564MiB:26%;cuda2:11435/24564MiB:89%260821-23:10:54 (UTC+8)
3443tasks done=4015/12141 running=2 failed=0 skipped=0 pending=8124 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:29995s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s3407:170s | GPU cuda1:7751/24564MiB:16%;cuda2:11435/24564MiB:53%260821-23:07:52 (UTC+8)
3442tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:29814s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:5149s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:99%260821-23:04:50 (UTC+8)
3441tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:29631s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:4966s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-23:01:49 (UTC+8)
3440tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:29450s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:4785s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260821-22:58:47 (UTC+8)
3439tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:29268s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:4603s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260821-22:55:45 (UTC+8)
3438tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:29085s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:4420s | GPU cuda1:7751/24564MiB:93%;cuda2:11435/24564MiB:100%260821-22:52:44 (UTC+8)
3437tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:28904s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:4239s | GPU cuda1:7751/24564MiB:17%;cuda2:11435/24564MiB:100%260821-22:49:40 (UTC+8)
3436tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:28721s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:4056s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260821-22:46:39 (UTC+8)
3435tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:28540s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:3875s | GPU cuda1:7751/24564MiB:42%;cuda2:11435/24564MiB:100%260821-22:43:36 (UTC+8)
3434tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:28358s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:3693s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-22:40:35 (UTC+8)
3433tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:28174s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:3509s | GPU cuda1:7751/24564MiB:25%;cuda2:11435/24564MiB:16%260821-22:37:33 (UTC+8)
3432tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:27991s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:3326s | GPU cuda1:7751/24564MiB:16%;cuda2:11435/24564MiB:100%260821-22:34:29 (UTC+8)
3431tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:27810s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:3144s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-22:31:26 (UTC+8)
3430tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:27628s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:2963s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:83%260821-22:28:25 (UTC+8)
3429tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:27446s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:2781s | GPU cuda1:7751/24564MiB:33%;cuda2:11435/24564MiB:100%260821-22:25:23 (UTC+8)
3428tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:27264s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:2599s | GPU cuda1:7751/24564MiB:34%;cuda2:11435/24564MiB:100%260821-22:22:21 (UTC+8)
3427tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:27082s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:2417s | GPU cuda1:7751/24564MiB:35%;cuda2:11435/24564MiB:100%260821-22:19:19 (UTC+8)
3426tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:26900s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:2235s | GPU cuda1:7751/24564MiB:31%;cuda2:11435/24564MiB:100%260821-22:16:18 (UTC+8)
3425tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:26719s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:2053s | GPU cuda1:7751/24564MiB:89%;cuda2:11435/24564MiB:100%260821-22:13:15 (UTC+8)
3424tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:26537s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:1872s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-22:10:14 (UTC+8)
3423tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:26356s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:1690s | GPU cuda1:7751/24564MiB:15%;cuda2:11435/24564MiB:58%260821-22:07:12 (UTC+8)
3422tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:26174s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:1509s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-22:04:10 (UTC+8)
3421tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:25992s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:1327s | GPU cuda1:7751/24564MiB:31%;cuda2:11435/24564MiB:85%260821-22:01:09 (UTC+8)
3420tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:25808s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:1143s | GPU cuda1:7751/24564MiB:28%;cuda2:11435/24564MiB:100%260821-21:58:07 (UTC+8)
3419tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:25626s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:961s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-21:55:03 (UTC+8)
3418tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:25445s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:780s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-21:52:01 (UTC+8)
3417tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:25263s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:598s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260821-21:49:00 (UTC+8)
3416tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:25080s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:415s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-21:45:58 (UTC+8)
3415tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:24898s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:233s | GPU cuda1:7751/24564MiB:75%;cuda2:11435/24564MiB:100%260821-21:42:55 (UTC+8)
3414tasks done=4013/12141 running=2 failed=0 skipped=0 pending=8126 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:24716s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s3407:51s | GPU cuda1:7751/24564MiB:35%;cuda2:11311/24564MiB:100%260821-21:39:53 (UTC+8)
3413tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:24534s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:5019s | GPU cuda1:7751/24564MiB:43%;cuda2:11435/24564MiB:100%260821-21:36:51 (UTC+8)
3412tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:24352s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:4837s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:100%260821-21:33:49 (UTC+8)
3411tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:24170s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:4655s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-21:30:47 (UTC+8)
3410tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:23988s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:4473s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-21:27:45 (UTC+8)
3409tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:23806s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:4291s | GPU cuda1:7751/24564MiB:41%;cuda2:11435/24564MiB:100%260821-21:24:43 (UTC+8)
3408tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:23624s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:4109s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:100%260821-21:21:41 (UTC+8)
3407tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:23442s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:3927s | GPU cuda1:7751/24564MiB:39%;cuda2:11435/24564MiB:100%260821-21:18:39 (UTC+8)
3406tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:23259s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:3745s | GPU cuda1:7751/24564MiB:28%;cuda2:11435/24564MiB:100%260821-21:15:37 (UTC+8)
3405tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:23078s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:3563s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:32%260821-21:12:35 (UTC+8)
3404tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:22896s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:3382s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-21:09:33 (UTC+8)
3403tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:22715s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:3200s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-21:06:32 (UTC+8)
3402tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:22533s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:3018s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-21:03:30 (UTC+8)
3401tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:22350s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:2835s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-21:00:28 (UTC+8)
3400tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:22168s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:2653s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:100%260821-20:57:25 (UTC+8)
3399tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:21986s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:2471s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-20:54:23 (UTC+8)
3398tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:21804s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:2290s | GPU cuda1:7751/24564MiB:37%;cuda2:11435/24564MiB:100%260821-20:51:21 (UTC+8)
3397tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:21623s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:2108s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-20:48:20 (UTC+8)
3396tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:21440s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:1925s | GPU cuda1:7751/24564MiB:31%;cuda2:11435/24564MiB:100%260821-20:45:18 (UTC+8)
3395tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:21258s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:1743s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-20:42:15 (UTC+8)
3394tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:21076s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:1561s | GPU cuda1:7751/24564MiB:48%;cuda2:11435/24564MiB:100%260821-20:39:13 (UTC+8)
3393tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:20894s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:1379s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-20:36:11 (UTC+8)
3392tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:20711s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:1197s | GPU cuda1:7751/24564MiB:39%;cuda2:11435/24564MiB:100%260821-20:33:09 (UTC+8)
3391tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:20530s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:1015s | GPU cuda1:7751/24564MiB:13%;cuda2:11435/24564MiB:100%260821-20:30:07 (UTC+8)
3390tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:20348s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:834s | GPU cuda1:7751/24564MiB:35%;cuda2:11435/24564MiB:100%260821-20:27:05 (UTC+8)
3389tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:20166s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:652s | GPU cuda1:7751/24564MiB:28%;cuda2:11435/24564MiB:100%260821-20:24:03 (UTC+8)
3388tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:19985s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:470s | GPU cuda1:7751/24564MiB:42%;cuda2:11435/24564MiB:100%260821-20:21:01 (UTC+8)
3387tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:19801s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:286s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-20:18:00 (UTC+8)
3386tasks done=4012/12141 running=2 failed=0 skipped=0 pending=8127 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:19619s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s3407:104s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-20:14:56 (UTC+8)
3385tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:19437s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:3329s | GPU cuda1:7751/24564MiB:26%;cuda2:11435/24564MiB:100%260821-20:11:54 (UTC+8)
3384tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:19256s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:3147s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-20:08:52 (UTC+8)
3383tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:19074s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:2966s | GPU cuda1:7751/24564MiB:25%;cuda2:11435/24564MiB:100%260821-20:05:51 (UTC+8)
3382tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:18893s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:2785s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260821-20:02:49 (UTC+8)
3381tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:18711s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:2603s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-19:59:48 (UTC+8)
3380tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:18529s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:2421s | GPU cuda1:7751/24564MiB:37%;cuda2:11435/24564MiB:100%260821-19:56:46 (UTC+8)
3379tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:18347s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:2239s | GPU cuda1:7751/24564MiB:36%;cuda2:11435/24564MiB:100%260821-19:53:44 (UTC+8)
3378tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:18164s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:2056s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:100%260821-19:50:42 (UTC+8)
3377tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:17982s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:1874s | GPU cuda1:7751/24564MiB:27%;cuda2:11435/24564MiB:100%260821-19:47:39 (UTC+8)
3376tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:17800s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:1692s | GPU cuda1:7751/24564MiB:31%;cuda2:11435/24564MiB:100%260821-19:44:37 (UTC+8)
3375tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:17619s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:1510s | GPU cuda1:7751/24564MiB:36%;cuda2:11435/24564MiB:100%260821-19:41:35 (UTC+8)
3374tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:17435s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:1327s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-19:38:34 (UTC+8)
3373tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:17252s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:1144s | GPU cuda1:7751/24564MiB:45%;cuda2:11435/24564MiB:100%260821-19:35:30 (UTC+8)
3372tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:17070s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:962s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-19:32:27 (UTC+8)
3371tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:16888s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:780s | GPU cuda1:7751/24564MiB:17%;cuda2:11435/24564MiB:100%260821-19:29:25 (UTC+8)
3370tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:16706s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:598s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:96%260821-19:26:23 (UTC+8)
3369tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:16525s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:417s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-19:23:21 (UTC+8)
3368tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:16343s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:235s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-19:20:20 (UTC+8)
3367tasks done=4010/12141 running=2 failed=0 skipped=0 pending=8129 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:16160s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s3407:52s | GPU cuda1:7751/24564MiB:19%;cuda2:11419/24564MiB:67%260821-19:17:18 (UTC+8)
3366tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:15978s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:3264s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-19:14:15 (UTC+8)
3365tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:15796s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:3082s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260821-19:11:13 (UTC+8)
3364tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:15613s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:2899s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:18%260821-19:08:11 (UTC+8)
3363tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:15430s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:2716s | GPU cuda1:7751/24564MiB:25%;cuda2:11435/24564MiB:100%260821-19:05:08 (UTC+8)
3362tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:15248s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:2535s | GPU cuda1:7751/24564MiB:28%;cuda2:11435/24564MiB:100%260821-19:02:05 (UTC+8)
3361tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:15067s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:2353s | GPU cuda1:7751/24564MiB:16%;cuda2:11435/24564MiB:100%260821-18:59:03 (UTC+8)
3360tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:14884s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:2171s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-18:56:02 (UTC+8)
3359tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:14702s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:1988s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:78%260821-18:52:59 (UTC+8)
3358tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:14520s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:1806s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:100%260821-18:49:57 (UTC+8)
3357tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:14338s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:1624s | GPU cuda1:7751/24564MiB:26%;cuda2:11435/24564MiB:69%260821-18:46:55 (UTC+8)
3356tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:14156s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:1442s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:39%260821-18:43:53 (UTC+8)
3355tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:13975s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:1261s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:34%260821-18:40:51 (UTC+8)
3354tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:13792s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:1079s | GPU cuda1:7751/24564MiB:26%;cuda2:11435/24564MiB:100%260821-18:37:50 (UTC+8)
3353tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:13610s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:897s | GPU cuda1:7751/24564MiB:38%;cuda2:11435/24564MiB:100%260821-18:34:47 (UTC+8)
3352tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:13428s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:715s | GPU cuda1:7751/24564MiB:46%;cuda2:11435/24564MiB:100%260821-18:31:45 (UTC+8)
3351tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:13247s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:533s | GPU cuda1:7751/24564MiB:15%;cuda2:11435/24564MiB:100%260821-18:28:44 (UTC+8)
3350tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:13065s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:351s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-18:25:42 (UTC+8)
3349tasks done=4009/12141 running=2 failed=0 skipped=0 pending=8130 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:12882s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s3407:168s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:94%260821-18:22:40 (UTC+8)
3348tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:12701s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:2189s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:89%260821-18:19:37 (UTC+8)
3347tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:12519s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:2007s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-18:16:36 (UTC+8)
3346tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:12336s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:1825s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260821-18:13:34 (UTC+8)
3345tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:12154s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:1643s | GPU cuda1:7751/24564MiB:92%;cuda2:11435/24564MiB:100%260821-18:10:32 (UTC+8)
3344tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:11972s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:1461s | GPU cuda1:7751/24564MiB:29%;cuda2:11435/24564MiB:87%260821-18:07:30 (UTC+8)
3343tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:11790s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:1279s | GPU cuda1:7751/24564MiB:16%;cuda2:11435/24564MiB:100%260821-18:04:27 (UTC+8)
3342tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:11608s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:1097s | GPU cuda1:7751/24564MiB:46%;cuda2:11435/24564MiB:98%260821-18:01:25 (UTC+8)
3341tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:11426s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:914s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-17:58:23 (UTC+8)
3340tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:11244s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:733s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-17:55:21 (UTC+8)
3339tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:11062s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:550s | GPU cuda1:7751/24564MiB:48%;cuda2:11435/24564MiB:100%260821-17:52:19 (UTC+8)
3338tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:10880s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:368s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-17:49:17 (UTC+8)
3337tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:10697s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:186s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:84%260821-17:46:15 (UTC+8)
3336tasks done=4007/12141 running=2 failed=0 skipped=0 pending=8132 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:10515s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s3407:3s | GPU cuda1:7751/24564MiB:20%;cuda2:3985/24564MiB:100%260821-17:43:12 (UTC+8)
3335tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:10333s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:2011s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:100%260821-17:40:10 (UTC+8)
3334tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:10151s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:1829s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-17:37:08 (UTC+8)
3333tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:9970s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:1648s | GPU cuda1:7751/24564MiB:17%;cuda2:11435/24564MiB:100%260821-17:34:06 (UTC+8)
3332tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:9788s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:1466s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-17:31:05 (UTC+8)
3331tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:9606s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:1284s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:79%260821-17:28:03 (UTC+8)
3330tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:9423s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:1101s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-17:25:01 (UTC+8)
3329tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:9241s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:919s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-17:21:58 (UTC+8)
3328tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:9059s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:737s | GPU cuda1:7751/24564MiB:33%;cuda2:11435/24564MiB:72%260821-17:18:56 (UTC+8)
3327tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:8877s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:555s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:49%260821-17:15:54 (UTC+8)
3326tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:8695s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:373s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:70%260821-17:12:52 (UTC+8)
3325tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:8513s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:191s | GPU cuda1:7751/24564MiB:33%;cuda2:11435/24564MiB:100%260821-17:09:50 (UTC+8)
3324tasks done=4006/12141 running=2 failed=0 skipped=0 pending=8133 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:8331s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s3407:9s | GPU cuda1:7751/24564MiB:21%;cuda2:3985/24564MiB:83%260821-17:06:48 (UTC+8)
3323tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:8149s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:4911s | GPU cuda1:7751/24564MiB:46%;cuda2:11435/24564MiB:100%260821-17:03:46 (UTC+8)
3322tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:7968s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:4729s | GPU cuda1:7751/24564MiB:17%;cuda2:11435/24564MiB:100%260821-17:00:44 (UTC+8)
3321tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:7785s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:4547s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260821-16:57:43 (UTC+8)
3320tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:7603s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:4365s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-16:54:40 (UTC+8)
3319tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:7421s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:4183s | GPU cuda1:7751/24564MiB:20%;cuda2:11435/24564MiB:100%260821-16:51:38 (UTC+8)
3318tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:7239s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:4001s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-16:48:36 (UTC+8)
3317tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:7057s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:3819s | GPU cuda1:7751/24564MiB:45%;cuda2:11435/24564MiB:98%260821-16:45:34 (UTC+8)
3316tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:6874s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:3635s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-16:42:32 (UTC+8)
3315tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:6692s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:3453s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-16:39:29 (UTC+8)
3314tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:6510s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:3272s | GPU cuda1:7751/24564MiB:40%;cuda2:11435/24564MiB:100%260821-16:36:27 (UTC+8)
3313tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:6328s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:3090s | GPU cuda1:7751/24564MiB:39%;cuda2:11435/24564MiB:100%260821-16:33:25 (UTC+8)
3312tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:6147s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:2909s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-16:30:23 (UTC+8)
3311tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:5965s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:2726s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-16:27:22 (UTC+8)
3310tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:5782s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:2543s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260821-16:24:20 (UTC+8)
3309tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:5600s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:2362s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-16:21:17 (UTC+8)
3308tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:5418s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:2179s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-16:18:15 (UTC+8)
3307tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:5236s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:1998s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-16:15:13 (UTC+8)
3306tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:5054s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:1815s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-16:12:11 (UTC+8)
3305tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:4872s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:1634s | GPU cuda1:7751/24564MiB:46%;cuda2:11435/24564MiB:100%260821-16:09:09 (UTC+8)
3304tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:4690s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:1452s | GPU cuda1:7751/24564MiB:29%;cuda2:11435/24564MiB:100%260821-16:06:07 (UTC+8)
3303tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:4509s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:1270s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-16:03:05 (UTC+8)
3302tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:4326s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:1087s | GPU cuda1:7751/24564MiB:18%;cuda2:11435/24564MiB:100%260821-16:00:04 (UTC+8)
3301tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:4143s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:904s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-15:57:01 (UTC+8)
3300tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:3961s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:722s | GPU cuda1:7751/24564MiB:17%;cuda2:11435/24564MiB:100%260821-15:53:58 (UTC+8)
3299tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:3778s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:539s | GPU cuda1:7751/24564MiB:31%;cuda2:11435/24564MiB:96%260821-15:50:56 (UTC+8)
3298tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:3596s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:358s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-15:47:53 (UTC+8)
3297tasks done=4004/12141 running=2 failed=0 skipped=0 pending=8135 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:3415s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s3407:176s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-15:44:51 (UTC+8)
3296tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:3233s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:5061s | GPU cuda1:7751/24564MiB:23%;cuda2:10935/24564MiB:98%260821-15:41:50 (UTC+8)
3295tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:3050s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:4878s | GPU cuda1:7751/24564MiB:16%;cuda2:11435/24564MiB:100%260821-15:38:48 (UTC+8)
3294tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:2868s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:4696s | GPU cuda1:7751/24564MiB:21%;cuda2:11435/24564MiB:100%260821-15:35:45 (UTC+8)
3293tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:2686s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:4514s | GPU cuda1:7751/24564MiB:19%;cuda2:11435/24564MiB:100%260821-15:32:43 (UTC+8)
3292tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:2504s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:4332s | GPU cuda1:7751/24564MiB:41%;cuda2:11435/24564MiB:100%260821-15:29:41 (UTC+8)
3291tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:2321s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:4148s | GPU cuda1:7751/24564MiB:11%;cuda2:11435/24564MiB:82%260821-15:26:39 (UTC+8)
3290tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:2139s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:3967s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-15:23:36 (UTC+8)
3289tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:1958s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:3785s | GPU cuda1:7751/24564MiB:26%;cuda2:11435/24564MiB:100%260821-15:20:35 (UTC+8)
3288tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:1776s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:3604s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:83%260821-15:17:33 (UTC+8)
3287tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:1595s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:3422s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260821-15:14:31 (UTC+8)
3286tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:1413s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:3241s | GPU cuda1:7751/24564MiB:27%;cuda2:11435/24564MiB:100%260821-15:11:30 (UTC+8)
3285tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:1231s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:3059s | GPU cuda1:7751/24564MiB:23%;cuda2:11435/24564MiB:100%260821-15:08:28 (UTC+8)
3284tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:1050s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:2877s | GPU cuda1:7751/24564MiB:17%;cuda2:11435/24564MiB:100%260821-15:05:27 (UTC+8)
3283tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:868s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:2696s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-15:02:25 (UTC+8)
3282tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:687s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:2514s | GPU cuda1:7751/24564MiB:24%;cuda2:11435/24564MiB:100%260821-14:59:23 (UTC+8)
3281tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:504s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:2332s | GPU cuda1:7751/24564MiB:16%;cuda2:11435/24564MiB:100%260821-14:56:22 (UTC+8)
3280tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:322s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:2150s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-14:53:19 (UTC+8)
3279tasks done=4003/12141 running=2 failed=0 skipped=0 pending=8136 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s999:141s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:1969s | GPU cuda1:7751/24564MiB:22%;cuda2:11435/24564MiB:100%260821-14:50:18 (UTC+8)
3278tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:2667s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:1785s | GPU cuda1:7851/24564MiB:91%;cuda2:11435/24564MiB:100%260821-14:47:16 (UTC+8)
3277tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:2485s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:1603s | GPU cuda1:7851/24564MiB:91%;cuda2:11435/24564MiB:100%260821-14:44:13 (UTC+8)
3276tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:2302s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:1421s | GPU cuda1:7851/24564MiB:21%;cuda2:11435/24564MiB:100%260821-14:41:11 (UTC+8)
3275tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:2120s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:1239s | GPU cuda1:7851/24564MiB:94%;cuda2:11435/24564MiB:100%260821-14:38:09 (UTC+8)
3274tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1938s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:1057s | GPU cuda1:7851/24564MiB:76%;cuda2:11435/24564MiB:100%260821-14:35:07 (UTC+8)
3273tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1756s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:875s | GPU cuda1:7851/24564MiB:94%;cuda2:11435/24564MiB:100%260821-14:32:04 (UTC+8)
3272tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1574s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:693s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-14:29:02 (UTC+8)
3271tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1392s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:511s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-14:26:00 (UTC+8)
3270tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1210s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:329s | GPU cuda1:7851/24564MiB:82%;cuda2:11435/24564MiB:100%260821-14:22:58 (UTC+8)
3269tasks done=4001/12141 running=2 failed=0 skipped=0 pending=8138 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1028s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s3407:147s | GPU cuda1:7851/24564MiB:88%;cuda2:11435/24564MiB:89%260821-14:19:56 (UTC+8)
3268tasks done=3999/12141 running=2 failed=0 skipped=0 pending=8140 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:846s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:3216s | GPU cuda1:7851/24564MiB:73%;cuda2:11435/24564MiB:100%260821-14:16:54 (UTC+8)
3267tasks done=3999/12141 running=2 failed=0 skipped=0 pending=8140 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:664s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:3034s | GPU cuda1:7851/24564MiB:93%;cuda2:11435/24564MiB:100%260821-14:13:52 (UTC+8)
3266tasks done=3999/12141 running=2 failed=0 skipped=0 pending=8140 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:481s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:2851s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-14:10:50 (UTC+8)
3265tasks done=3999/12141 running=2 failed=0 skipped=0 pending=8140 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:300s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:2669s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-14:07:48 (UTC+8)
3264tasks done=3999/12141 running=2 failed=0 skipped=0 pending=8140 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s999:118s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:2488s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-14:04:46 (UTC+8)
3263tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:7021s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:2305s | GPU cuda1:11839/24564MiB:97%;cuda2:11435/24564MiB:100%260821-14:01:44 (UTC+8)
3262tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:6839s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:2123s | GPU cuda1:11839/24564MiB:99%;cuda2:11435/24564MiB:100%260821-13:58:41 (UTC+8)
3261tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:6658s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:1941s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:55:39 (UTC+8)
3260tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:6476s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:1759s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:52:38 (UTC+8)
3259tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:6294s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:1577s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:49:36 (UTC+8)
3258tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:6112s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:1396s | GPU cuda1:11839/24564MiB:90%;cuda2:11435/24564MiB:89%260821-13:46:34 (UTC+8)
3257tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:5930s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:1213s | GPU cuda1:11839/24564MiB:38%;cuda2:11435/24564MiB:100%260821-13:43:32 (UTC+8)
3256tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:5748s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:1032s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:40:30 (UTC+8)
3255tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:5566s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:850s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:49%260821-13:37:28 (UTC+8)
3254tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:5385s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:668s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:86%260821-13:34:26 (UTC+8)
3253tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:5203s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:487s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:31:24 (UTC+8)
3252tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:5019s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:302s | GPU cuda1:11839/24564MiB:80%;cuda2:11435/24564MiB:86%260821-13:28:23 (UTC+8)
3251tasks done=3998/12141 running=2 failed=0 skipped=0 pending=8141 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:4835s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s3407:119s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:25:18 (UTC+8)
3250tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:4653s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:3255s | GPU cuda1:11839/24564MiB:33%;cuda2:11435/24564MiB:68%260821-13:22:15 (UTC+8)
3249tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:4471s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:3073s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:19:13 (UTC+8)
3248tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:4289s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:2891s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:16:11 (UTC+8)
3247tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:4107s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:2709s | GPU cuda1:11839/24564MiB:92%;cuda2:11435/24564MiB:100%260821-13:13:09 (UTC+8)
3246tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:3926s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:2527s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:90%260821-13:10:07 (UTC+8)
3245tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:3744s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:2345s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:07:05 (UTC+8)
3244tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:3562s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:2164s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:04:04 (UTC+8)
3243tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:3380s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:1982s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-13:01:02 (UTC+8)
3242tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:3196s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:1798s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-12:58:00 (UTC+8)
3241tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:3014s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:1616s | GPU cuda1:11839/24564MiB:83%;cuda2:11435/24564MiB:100%260821-12:54:56 (UTC+8)
3240tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:2833s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:1434s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-12:51:54 (UTC+8)
3239tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:2651s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:1252s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-12:48:52 (UTC+8)
3238tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:2467s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:1069s | GPU cuda1:11839/24564MiB:87%;cuda2:11435/24564MiB:87%260821-12:45:51 (UTC+8)
3237tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:2286s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:887s | GPU cuda1:11839/24564MiB:86%;cuda2:11435/24564MiB:100%260821-12:42:47 (UTC+8)
3236tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:2103s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:704s | GPU cuda1:11839/24564MiB:98%;cuda2:11435/24564MiB:100%260821-12:39:46 (UTC+8)
3235tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:1921s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:522s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-12:36:42 (UTC+8)
3234tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:1738s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:340s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-12:33:40 (UTC+8)
3233tasks done=3997/12141 running=2 failed=0 skipped=0 pending=8142 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:1557s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s3407:158s | GPU cuda1:11839/24564MiB:89%;cuda2:11435/24564MiB:100%260821-12:30:38 (UTC+8)
3232tasks done=3995/12141 running=2 failed=0 skipped=0 pending=8144 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:1374s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:2469s | GPU cuda1:11839/24564MiB:97%;cuda2:11435/24564MiB:79%260821-12:27:37 (UTC+8)
3231tasks done=3995/12141 running=2 failed=0 skipped=0 pending=8144 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:1191s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:2286s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-12:24:33 (UTC+8)
3230tasks done=3995/12141 running=2 failed=0 skipped=0 pending=8144 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:1009s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:2104s | GPU cuda1:11839/24564MiB:88%;cuda2:11435/24564MiB:98%260821-12:21:31 (UTC+8)
3229tasks done=3995/12141 running=2 failed=0 skipped=0 pending=8144 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:828s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:1923s | GPU cuda1:11839/24564MiB:97%;cuda2:11435/24564MiB:100%260821-12:18:29 (UTC+8)
3228tasks done=3995/12141 running=2 failed=0 skipped=0 pending=8144 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:646s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:1741s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-12:15:28 (UTC+8)
3227tasks done=3995/12141 running=2 failed=0 skipped=0 pending=8144 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:464s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:1559s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260821-12:12:26 (UTC+8)
3226tasks done=3995/12141 running=2 failed=0 skipped=0 pending=8144 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:282s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:1377s | GPU cuda1:7649/24564MiB:20%;cuda2:11435/24564MiB:100%260821-12:09:24 (UTC+8)
3225tasks done=3995/12141 running=2 failed=0 skipped=0 pending=8144 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s999:100s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:1195s | GPU cuda1:401/24564MiB:0%;cuda2:11435/24564MiB:100%260821-12:06:22 (UTC+8)
3224tasks done=3993/12141 running=2 failed=0 skipped=0 pending=8146 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:2350s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:1013s | GPU cuda1:7851/24564MiB:94%;cuda2:11435/24564MiB:100%260821-12:03:20 (UTC+8)
3223tasks done=3993/12141 running=2 failed=0 skipped=0 pending=8146 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:2168s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:831s | GPU cuda1:7851/24564MiB:79%;cuda2:11435/24564MiB:100%260821-12:00:18 (UTC+8)
3222tasks done=3993/12141 running=2 failed=0 skipped=0 pending=8146 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:1986s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:649s | GPU cuda1:7851/24564MiB:85%;cuda2:11435/24564MiB:100%260821-11:57:16 (UTC+8)
3221tasks done=3993/12141 running=2 failed=0 skipped=0 pending=8146 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:1804s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:467s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-11:54:14 (UTC+8)
3220tasks done=3993/12141 running=2 failed=0 skipped=0 pending=8146 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:1622s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:285s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-11:51:12 (UTC+8)
3219tasks done=3993/12141 running=2 failed=0 skipped=0 pending=8146 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:1440s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s3407:103s | GPU cuda1:7851/24564MiB:90%;cuda2:11435/24564MiB:100%260821-11:48:10 (UTC+8)
3218tasks done=3992/12141 running=2 failed=0 skipped=0 pending=8147 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:1258s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:2410s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260821-11:45:08 (UTC+8)
3217tasks done=3992/12141 running=2 failed=0 skipped=0 pending=8147 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:1076s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:2227s | GPU cuda1:7851/24564MiB:93%;cuda2:11435/24564MiB:100%260821-11:42:06 (UTC+8)
3216tasks done=3992/12141 running=2 failed=0 skipped=0 pending=8147 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:894s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:2045s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-11:39:04 (UTC+8)
3215tasks done=3992/12141 running=2 failed=0 skipped=0 pending=8147 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:713s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:1864s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-11:36:02 (UTC+8)
3214tasks done=3992/12141 running=2 failed=0 skipped=0 pending=8147 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:530s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:1682s | GPU cuda1:7851/24564MiB:92%;cuda2:11435/24564MiB:49%260821-11:33:00 (UTC+8)
3213tasks done=3992/12141 running=2 failed=0 skipped=0 pending=8147 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:348s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:1500s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-11:29:58 (UTC+8)
3212tasks done=3992/12141 running=2 failed=0 skipped=0 pending=8147 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s999:167s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:1318s | GPU cuda1:7851/24564MiB:54%;cuda2:11435/24564MiB:76%260821-11:26:56 (UTC+8)
3211tasks done=3991/12141 running=2 failed=0 skipped=0 pending=8148 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:2431s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:1137s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-11:23:55 (UTC+8)
3210tasks done=3991/12141 running=2 failed=0 skipped=0 pending=8148 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:2248s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:954s | GPU cuda1:7851/24564MiB:87%;cuda2:11435/24564MiB:84%260821-11:20:53 (UTC+8)
3209tasks done=3991/12141 running=2 failed=0 skipped=0 pending=8148 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:2066s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:772s | GPU cuda1:7851/24564MiB:85%;cuda2:11435/24564MiB:100%260821-11:17:51 (UTC+8)
3208tasks done=3991/12141 running=2 failed=0 skipped=0 pending=8148 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:1885s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:590s | GPU cuda1:7851/24564MiB:90%;cuda2:11435/24564MiB:100%260821-11:14:49 (UTC+8)
3207tasks done=3991/12141 running=2 failed=0 skipped=0 pending=8148 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:1703s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:409s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-11:11:47 (UTC+8)
3206tasks done=3991/12141 running=2 failed=0 skipped=0 pending=8148 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:1521s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:227s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-11:08:45 (UTC+8)
3205tasks done=3991/12141 running=2 failed=0 skipped=0 pending=8148 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:1339s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s3407:45s | GPU cuda1:7851/24564MiB:70%;cuda2:11297/24564MiB:85%260821-11:05:43 (UTC+8)
3204tasks done=3989/12141 running=2 failed=0 skipped=0 pending=8150 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:1156s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:1976s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-11:02:42 (UTC+8)
3203tasks done=3989/12141 running=2 failed=0 skipped=0 pending=8150 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:975s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:1795s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-10:59:39 (UTC+8)
3202tasks done=3989/12141 running=2 failed=0 skipped=0 pending=8150 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:792s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:1612s | GPU cuda1:7851/24564MiB:64%;cuda2:11435/24564MiB:81%260821-10:56:37 (UTC+8)
3201tasks done=3989/12141 running=2 failed=0 skipped=0 pending=8150 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:611s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:1431s | GPU cuda1:7851/24564MiB:92%;cuda2:11435/24564MiB:100%260821-10:53:35 (UTC+8)
3200tasks done=3989/12141 running=2 failed=0 skipped=0 pending=8150 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:429s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:1249s | GPU cuda1:7851/24564MiB:86%;cuda2:11435/24564MiB:100%260821-10:50:34 (UTC+8)
3199tasks done=3989/12141 running=2 failed=0 skipped=0 pending=8150 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:247s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:1067s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-10:47:32 (UTC+8)
3198tasks done=3989/12141 running=2 failed=0 skipped=0 pending=8150 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s999:66s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:886s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-10:44:30 (UTC+8)
3197tasks done=3987/12141 running=2 failed=0 skipped=0 pending=8152 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s999:1569s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:704s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-10:41:29 (UTC+8)
3196tasks done=3987/12141 running=2 failed=0 skipped=0 pending=8152 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s999:1387s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:522s | GPU cuda1:7851/24564MiB:87%;cuda2:11435/24564MiB:100%260821-10:38:26 (UTC+8)
3195tasks done=3987/12141 running=2 failed=0 skipped=0 pending=8152 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s999:1205s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:341s | GPU cuda1:7851/24564MiB:86%;cuda2:11435/24564MiB:100%260821-10:35:25 (UTC+8)
3194tasks done=3987/12141 running=2 failed=0 skipped=0 pending=8152 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s999:1024s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s3407:159s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-10:32:23 (UTC+8)
3193tasks done=3986/12141 running=2 failed=0 skipped=0 pending=8153 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s999:842s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:2017s | GPU cuda1:7851/24564MiB:73%;cuda2:11435/24564MiB:100%260821-10:29:22 (UTC+8)
3192tasks done=3986/12141 running=2 failed=0 skipped=0 pending=8153 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s999:660s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:1835s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-10:26:20 (UTC+8)
3191tasks done=3986/12141 running=2 failed=0 skipped=0 pending=8153 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s999:479s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:1653s | GPU cuda1:7851/24564MiB:86%;cuda2:11435/24564MiB:100%260821-10:23:18 (UTC+8)
3190tasks done=3986/12141 running=2 failed=0 skipped=0 pending=8153 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s999:298s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:1472s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-10:20:17 (UTC+8)
3189tasks done=3986/12141 running=2 failed=0 skipped=0 pending=8153 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s999:116s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:1290s | GPU cuda1:7851/24564MiB:77%;cuda2:11435/24564MiB:96%260821-10:17:16 (UTC+8)
3188tasks done=3985/12141 running=2 failed=0 skipped=0 pending=8154 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s999:1366s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:1108s | GPU cuda1:7851/24564MiB:86%;cuda2:11435/24564MiB:100%260821-10:14:14 (UTC+8)
3187tasks done=3985/12141 running=2 failed=0 skipped=0 pending=8154 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s999:1183s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:925s | GPU cuda1:7851/24564MiB:93%;cuda2:11445/24564MiB:100%260821-10:11:12 (UTC+8)
3186tasks done=3985/12141 running=2 failed=0 skipped=0 pending=8154 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s999:1001s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:743s | GPU cuda1:7851/24564MiB:100%;cuda2:11445/24564MiB:100%260821-10:08:09 (UTC+8)
3185tasks done=3985/12141 running=2 failed=0 skipped=0 pending=8154 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s999:820s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:562s | GPU cuda1:7851/24564MiB:97%;cuda2:11445/24564MiB:100%260821-10:05:07 (UTC+8)
3184tasks done=3985/12141 running=2 failed=0 skipped=0 pending=8154 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s999:639s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:381s | GPU cuda1:7851/24564MiB:97%;cuda2:11445/24564MiB:100%260821-10:02:06 (UTC+8)
3183tasks done=3985/12141 running=2 failed=0 skipped=0 pending=8154 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s999:457s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:200s | GPU cuda1:7851/24564MiB:96%;cuda2:11445/24564MiB:100%260821-09:59:05 (UTC+8)
3182tasks done=3985/12141 running=2 failed=0 skipped=0 pending=8154 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s999:276s;cuda2:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s3407:19s | GPU cuda1:7851/24564MiB:97%;cuda2:11299/24564MiB:100%260821-09:56:04 (UTC+8)
3181tasks done=3983/12141 running=2 failed=0 skipped=0 pending=8156 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s999:95s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:5078s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-09:53:03 (UTC+8)
3180tasks done=3981/12141 running=2 failed=0 skipped=0 pending=8158 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s999:838s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:4896s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-09:50:02 (UTC+8)
3179tasks done=3981/12141 running=2 failed=0 skipped=0 pending=8158 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s999:656s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:4715s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-09:47:00 (UTC+8)
3178tasks done=3981/12141 running=2 failed=0 skipped=0 pending=8158 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s999:475s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:4534s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-09:43:59 (UTC+8)
3177tasks done=3981/12141 running=2 failed=0 skipped=0 pending=8158 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s999:293s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:4352s | GPU cuda1:7851/24564MiB:25%;cuda2:11435/24564MiB:100%260821-09:40:57 (UTC+8)
3176tasks done=3981/12141 running=2 failed=0 skipped=0 pending=8158 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s999:112s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:4171s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-09:37:56 (UTC+8)
3175tasks done=3980/12141 running=2 failed=0 skipped=0 pending=8159 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s999:852s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:3990s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-09:34:55 (UTC+8)
3174tasks done=3980/12141 running=2 failed=0 skipped=0 pending=8159 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s999:670s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:3809s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-09:31:54 (UTC+8)
3173tasks done=3980/12141 running=2 failed=0 skipped=0 pending=8159 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s999:489s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:3627s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-09:28:52 (UTC+8)
3172tasks done=3980/12141 running=2 failed=0 skipped=0 pending=8159 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s999:308s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:3446s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-09:25:51 (UTC+8)
3171tasks done=3980/12141 running=2 failed=0 skipped=0 pending=8159 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s999:127s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:3265s | GPU cuda1:7851/24564MiB:9%;cuda2:11435/24564MiB:100%260821-09:22:50 (UTC+8)
3170tasks done=3978/12141 running=2 failed=0 skipped=0 pending=8161 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s999:744s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:3084s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-09:19:49 (UTC+8)
3169tasks done=3978/12141 running=2 failed=0 skipped=0 pending=8161 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s999:562s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:2903s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260821-09:16:48 (UTC+8)
3168tasks done=3978/12141 running=2 failed=0 skipped=0 pending=8161 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s999:381s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:2722s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260821-09:13:46 (UTC+8)
3167tasks done=3978/12141 running=2 failed=0 skipped=0 pending=8161 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s999:200s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:2541s | GPU cuda1:7851/24564MiB:89%;cuda2:11435/24564MiB:100%260821-09:10:45 (UTC+8)
3166tasks done=3978/12141 running=2 failed=0 skipped=0 pending=8161 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s999:19s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:2360s | GPU cuda1:7713/24564MiB:23%;cuda2:11435/24564MiB:100%260821-09:07:44 (UTC+8)
3165tasks done=3977/12141 running=2 failed=0 skipped=0 pending=8162 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s999:633s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:2178s | GPU cuda1:7851/24564MiB:94%;cuda2:11435/24564MiB:100%260821-09:04:43 (UTC+8)
3164tasks done=3977/12141 running=2 failed=0 skipped=0 pending=8162 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s999:452s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:1997s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260821-09:01:42 (UTC+8)
3163tasks done=3977/12141 running=2 failed=0 skipped=0 pending=8162 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s999:271s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:1816s | GPU cuda1:7851/24564MiB:93%;cuda2:11435/24564MiB:100%260821-08:58:41 (UTC+8)
3162tasks done=3977/12141 running=2 failed=0 skipped=0 pending=8162 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s999:90s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:1635s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-08:55:40 (UTC+8)
3161tasks done=3975/12141 running=2 failed=0 skipped=0 pending=8164 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:4364s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:1453s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-08:52:39 (UTC+8)
3160tasks done=3975/12141 running=2 failed=0 skipped=0 pending=8164 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:4183s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:1272s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-08:49:37 (UTC+8)
3159tasks done=3975/12141 running=2 failed=0 skipped=0 pending=8164 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:4002s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:1091s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:46:36 (UTC+8)
3158tasks done=3975/12141 running=2 failed=0 skipped=0 pending=8164 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:3820s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:910s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:43:35 (UTC+8)
3157tasks done=3975/12141 running=2 failed=0 skipped=0 pending=8164 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:3639s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:729s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-08:40:33 (UTC+8)
3156tasks done=3975/12141 running=2 failed=0 skipped=0 pending=8164 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:3458s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:547s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:37:32 (UTC+8)
3155tasks done=3975/12141 running=2 failed=0 skipped=0 pending=8164 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:3276s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:366s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:34:31 (UTC+8)
3154tasks done=3975/12141 running=2 failed=0 skipped=0 pending=8164 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:3095s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:185s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-08:31:29 (UTC+8)
3153tasks done=3975/12141 running=2 failed=0 skipped=0 pending=8164 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:2914s;cuda2:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s3407:4s | GPU cuda1:7851/24564MiB:95%;cuda2:3985/24564MiB:100%260821-08:28:28 (UTC+8)
3152tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:2733s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:5060s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:25:27 (UTC+8)
3151tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:2551s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:4878s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:22:26 (UTC+8)
3150tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:2370s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:4697s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-08:19:24 (UTC+8)
3149tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:2189s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:4516s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-08:16:23 (UTC+8)
3148tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:2008s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:4335s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:13:22 (UTC+8)
3147tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:1827s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:4154s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:10:21 (UTC+8)
3146tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:1645s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:3973s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:07:20 (UTC+8)
3145tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:1464s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:3792s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-08:04:19 (UTC+8)
3144tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:1283s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:3611s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-08:01:18 (UTC+8)
3143tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:1102s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:3430s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:58:17 (UTC+8)
3142tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:920s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:3248s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:55:16 (UTC+8)
3141tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:739s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:3066s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-07:52:14 (UTC+8)
3140tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:558s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:2885s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:49:12 (UTC+8)
3139tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:376s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:2703s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:46:11 (UTC+8)
3138tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:195s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:2522s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:43:09 (UTC+8)
3137tasks done=3974/12141 running=2 failed=0 skipped=0 pending=8165 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s999:14s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:2341s | GPU cuda1:7679/24564MiB:21%;cuda2:11435/24564MiB:100%260821-07:40:08 (UTC+8)
3136tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:4345s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:2160s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:37:07 (UTC+8)
3135tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:4162s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:1977s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:34:06 (UTC+8)
3134tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:3981s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:1796s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-07:31:03 (UTC+8)
3133tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:3800s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:1615s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:28:02 (UTC+8)
3132tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:3619s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:1434s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:25:01 (UTC+8)
3131tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:3437s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:1253s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-07:22:00 (UTC+8)
3130tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:3256s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:1071s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:18:59 (UTC+8)
3129tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:3074s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:890s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:15:57 (UTC+8)
3128tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:2893s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:709s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-07:12:56 (UTC+8)
3127tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:2711s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:527s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-07:09:55 (UTC+8)
3126tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:2530s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:346s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:06:53 (UTC+8)
3125tasks done=3973/12141 running=2 failed=0 skipped=0 pending=8166 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:2349s;cuda2:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s3407:164s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-07:03:52 (UTC+8)
3124tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:2168s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:2356s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-07:00:50 (UTC+8)
3123tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:1987s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:2175s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-06:57:49 (UTC+8)
3122tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:1805s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:1994s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:54:48 (UTC+8)
3121tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:1624s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:1813s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-06:51:47 (UTC+8)
3120tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:1443s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:1632s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-06:48:45 (UTC+8)
3119tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:1262s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:1450s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:45:44 (UTC+8)
3118tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:1080s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:1269s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:42:43 (UTC+8)
3117tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:899s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:1088s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-06:39:42 (UTC+8)
3116tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:718s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:907s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:36:41 (UTC+8)
3115tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:537s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:726s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:33:40 (UTC+8)
3114tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:356s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:544s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-06:30:39 (UTC+8)
3113tasks done=3971/12141 running=2 failed=0 skipped=0 pending=8168 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s999:174s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:363s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:27:37 (UTC+8)
3112tasks done=3969/12141 running=2 failed=0 skipped=0 pending=8170 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:3594s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:182s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-06:24:36 (UTC+8)
3111tasks done=3969/12141 running=2 failed=0 skipped=0 pending=8170 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:3413s;cuda2:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s3407:1s | GPU cuda1:7851/24564MiB:97%;cuda2:3597/24564MiB:100%260821-06:21:34 (UTC+8)
3110tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:3231s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:2190s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:18:33 (UTC+8)
3109tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:3050s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:2009s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:15:32 (UTC+8)
3108tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:2869s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:1828s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-06:12:31 (UTC+8)
3107tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:2688s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:1647s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:09:30 (UTC+8)
3106tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:2507s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:1465s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-06:06:29 (UTC+8)
3105tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:2326s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:1284s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:03:28 (UTC+8)
3104tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:2145s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:1103s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-06:00:27 (UTC+8)
3103tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:1963s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:922s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-05:57:25 (UTC+8)
3102tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:1782s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:741s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-05:54:24 (UTC+8)
3101tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:1601s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:560s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:51:23 (UTC+8)
3100tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:1420s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:378s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:48:22 (UTC+8)
3099tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:1239s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:197s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-05:45:20 (UTC+8)
3098tasks done=3968/12141 running=2 failed=0 skipped=0 pending=8171 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:1057s;cuda2:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s3407:16s | GPU cuda1:7851/24564MiB:0%;cuda2:11271/24564MiB:100%260821-05:42:19 (UTC+8)
3097tasks done=3966/12141 running=2 failed=0 skipped=0 pending=8173 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:876s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:1808s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-05:39:18 (UTC+8)
3096tasks done=3966/12141 running=2 failed=0 skipped=0 pending=8173 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:695s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:1627s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:36:17 (UTC+8)
3095tasks done=3966/12141 running=2 failed=0 skipped=0 pending=8173 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:514s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:1446s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:33:16 (UTC+8)
3094tasks done=3966/12141 running=2 failed=0 skipped=0 pending=8173 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:333s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:1264s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-05:30:15 (UTC+8)
3093tasks done=3966/12141 running=2 failed=0 skipped=0 pending=8173 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s999:151s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:1082s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:27:13 (UTC+8)
3092tasks done=3965/12141 running=2 failed=0 skipped=0 pending=8174 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:3571s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:901s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-05:24:12 (UTC+8)
3091tasks done=3965/12141 running=2 failed=0 skipped=0 pending=8174 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:3390s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:720s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:21:10 (UTC+8)
3090tasks done=3965/12141 running=2 failed=0 skipped=0 pending=8174 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:3209s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:539s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:18:09 (UTC+8)
3089tasks done=3965/12141 running=2 failed=0 skipped=0 pending=8174 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:3028s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:358s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-05:15:08 (UTC+8)
3088tasks done=3965/12141 running=2 failed=0 skipped=0 pending=8174 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:2847s;cuda2:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s3407:177s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:12:07 (UTC+8)
3087tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:2666s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:1967s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-05:09:06 (UTC+8)
3086tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:2485s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:1786s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:06:05 (UTC+8)
3085tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:2304s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:1604s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-05:03:04 (UTC+8)
3084tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:2122s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:1423s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-05:00:03 (UTC+8)
3083tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:1936s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:1237s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:57:02 (UTC+8)
3082tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:1755s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:1056s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:53:55 (UTC+8)
3081tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:1574s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:874s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260821-04:50:54 (UTC+8)
3080tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:1393s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:693s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-04:47:53 (UTC+8)
3079tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:1212s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:512s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:44:52 (UTC+8)
3078tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:1031s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:331s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-04:41:51 (UTC+8)
3077tasks done=3964/12141 running=2 failed=0 skipped=0 pending=8175 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:850s;cuda2:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s3407:150s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-04:38:50 (UTC+8)
3076tasks done=3962/12141 running=2 failed=0 skipped=0 pending=8177 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:669s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:2374s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:35:49 (UTC+8)
3075tasks done=3962/12141 running=2 failed=0 skipped=0 pending=8177 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:488s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:2193s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-04:32:48 (UTC+8)
3074tasks done=3962/12141 running=2 failed=0 skipped=0 pending=8177 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:307s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:2012s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:29:47 (UTC+8)
3073tasks done=3962/12141 running=2 failed=0 skipped=0 pending=8177 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s999:125s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:1831s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:26:46 (UTC+8)
3072tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:3143s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:1649s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-04:23:44 (UTC+8)
3071tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:2961s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:1468s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:20:43 (UTC+8)
3070tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:2780s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:1286s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:17:41 (UTC+8)
3069tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:2599s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:1105s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:14:40 (UTC+8)
3068tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:2418s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:924s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:11:39 (UTC+8)
3067tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:2237s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:743s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-04:08:38 (UTC+8)
3066tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:2056s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:562s | GPU cuda1:7851/24564MiB:63%;cuda2:11435/24564MiB:100%260821-04:05:37 (UTC+8)
3065tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:1875s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:381s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-04:02:36 (UTC+8)
3064tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:1694s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:200s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:59:35 (UTC+8)
3063tasks done=3960/12141 running=2 failed=0 skipped=0 pending=8179 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:1512s;cuda2:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s3407:19s | GPU cuda1:7851/24564MiB:97%;cuda2:11293/24564MiB:100%260821-03:56:34 (UTC+8)
3062tasks done=3959/12141 running=2 failed=0 skipped=0 pending=8180 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:1331s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:2239s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:53:32 (UTC+8)
3061tasks done=3959/12141 running=2 failed=0 skipped=0 pending=8180 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:1150s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:2058s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:50:30 (UTC+8)
3060tasks done=3959/12141 running=2 failed=0 skipped=0 pending=8180 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:969s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:1877s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:47:29 (UTC+8)
3059tasks done=3959/12141 running=2 failed=0 skipped=0 pending=8180 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:788s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:1696s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-03:44:28 (UTC+8)
3058tasks done=3959/12141 running=2 failed=0 skipped=0 pending=8180 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:606s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:1515s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-03:41:27 (UTC+8)
3057tasks done=3959/12141 running=2 failed=0 skipped=0 pending=8180 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:425s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:1334s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:38:26 (UTC+8)
3056tasks done=3959/12141 running=2 failed=0 skipped=0 pending=8180 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:244s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:1153s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:35:25 (UTC+8)
3055tasks done=3959/12141 running=2 failed=0 skipped=0 pending=8180 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s999:63s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:972s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:32:24 (UTC+8)
3054tasks done=3958/12141 running=2 failed=0 skipped=0 pending=8181 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:3080s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:790s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:29:23 (UTC+8)
3053tasks done=3958/12141 running=2 failed=0 skipped=0 pending=8181 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:2898s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:609s | GPU cuda1:7851/24564MiB:94%;cuda2:11435/24564MiB:100%260821-03:26:21 (UTC+8)
3052tasks done=3958/12141 running=2 failed=0 skipped=0 pending=8181 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:2717s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:428s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:23:20 (UTC+8)
3051tasks done=3958/12141 running=2 failed=0 skipped=0 pending=8181 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:2536s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:247s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:20:19 (UTC+8)
3050tasks done=3958/12141 running=2 failed=0 skipped=0 pending=8181 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:2355s;cuda2:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s3407:66s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:17:18 (UTC+8)
3049tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:2174s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:3177s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:14:17 (UTC+8)
3048tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:1993s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:2996s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-03:11:16 (UTC+8)
3047tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:1812s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:2815s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:08:14 (UTC+8)
3046tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:1631s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:2633s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-03:05:13 (UTC+8)
3045tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:1450s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:2452s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-03:02:12 (UTC+8)
3044tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:1268s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:2271s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:59:11 (UTC+8)
3043tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:1087s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:2090s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:56:10 (UTC+8)
3042tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:906s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:1909s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:53:09 (UTC+8)
3041tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:725s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:1728s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-02:50:08 (UTC+8)
3040tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:544s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:1547s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-02:47:07 (UTC+8)
3039tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:361s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:1364s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:44:05 (UTC+8)
3038tasks done=3956/12141 running=2 failed=0 skipped=0 pending=8183 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s999:179s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:1182s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:41:02 (UTC+8)
3037tasks done=3955/12141 running=2 failed=0 skipped=0 pending=8184 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:pair_audit:pair_audit__Diverse-128__llama3_8b__wikics__s999:1s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:1001s | GPU cuda1:14/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:38:01 (UTC+8)
3036tasks done=3954/12141 running=2 failed=0 skipped=0 pending=8185 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:2931s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:820s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:35:00 (UTC+8)
3035tasks done=3954/12141 running=2 failed=0 skipped=0 pending=8185 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:2750s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:639s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-02:31:59 (UTC+8)
3034tasks done=3954/12141 running=2 failed=0 skipped=0 pending=8185 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:2568s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:457s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:28:58 (UTC+8)
3033tasks done=3954/12141 running=2 failed=0 skipped=0 pending=8185 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:2387s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:276s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:25:56 (UTC+8)
3032tasks done=3954/12141 running=2 failed=0 skipped=0 pending=8185 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:2206s;cuda2:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s3407:95s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260821-02:22:55 (UTC+8)
3031tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:2025s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:3204s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:19:54 (UTC+8)
3030tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:1844s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:3023s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:16:53 (UTC+8)
3029tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:1662s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:2841s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:13:52 (UTC+8)
3028tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:1481s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:2660s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:10:50 (UTC+8)
3027tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:1300s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:2479s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:07:49 (UTC+8)
3026tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:1118s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:2298s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-02:04:48 (UTC+8)
3025tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:937s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:2117s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260821-02:01:47 (UTC+8)
3024tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:756s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:1936s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:58:46 (UTC+8)
3023tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:575s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:1755s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:55:45 (UTC+8)
3022tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:394s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:1573s | GPU cuda1:7851/24564MiB:92%;cuda2:11435/24564MiB:100%260821-01:52:43 (UTC+8)
3021tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:213s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:1392s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:49:42 (UTC+8)
3020tasks done=3953/12141 running=2 failed=0 skipped=0 pending=8186 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s999:31s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:1210s | GPU cuda1:7835/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:46:41 (UTC+8)
3019tasks done=3952/12141 running=2 failed=0 skipped=0 pending=8187 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:2867s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:1029s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-01:43:39 (UTC+8)
3018tasks done=3952/12141 running=2 failed=0 skipped=0 pending=8187 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:2685s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:848s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:40:38 (UTC+8)
3017tasks done=3952/12141 running=2 failed=0 skipped=0 pending=8187 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:2504s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:667s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260821-01:37:37 (UTC+8)
3016tasks done=3952/12141 running=2 failed=0 skipped=0 pending=8187 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:2322s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:485s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:34:35 (UTC+8)
3015tasks done=3952/12141 running=2 failed=0 skipped=0 pending=8187 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:2141s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:304s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-01:31:34 (UTC+8)
3014tasks done=3952/12141 running=2 failed=0 skipped=0 pending=8187 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:1960s;cuda2:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s3407:123s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:28:32 (UTC+8)
3013tasks done=3951/12141 running=2 failed=0 skipped=0 pending=8188 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:1779s;cuda2:trace_context:trace_bank__Self__llama3_8b__wikics__s3407:1572s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260821-01:25:31 (UTC+8)
3012tasks done=3951/12141 running=2 failed=0 skipped=0 pending=8188 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:1598s;cuda2:trace_context:trace_bank__Self__llama3_8b__wikics__s3407:1390s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:22:30 (UTC+8)
3011tasks done=3951/12141 running=2 failed=0 skipped=0 pending=8188 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:1417s;cuda2:trace_context:trace_bank__Self__llama3_8b__wikics__s3407:1209s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:19:29 (UTC+8)
3010tasks done=3951/12141 running=2 failed=0 skipped=0 pending=8188 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:1236s;cuda2:trace_context:trace_bank__Self__llama3_8b__wikics__s3407:1028s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260821-01:16:28 (UTC+8)
3009tasks done=3951/12141 running=2 failed=0 skipped=0 pending=8188 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:1055s;cuda2:trace_context:trace_bank__Self__llama3_8b__wikics__s3407:847s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:13:27 (UTC+8)
3008tasks done=3951/12141 running=2 failed=0 skipped=0 pending=8188 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:874s;cuda2:trace_context:trace_bank__Self__llama3_8b__wikics__s3407:666s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-01:10:26 (UTC+8)
3007tasks done=3951/12141 running=2 failed=0 skipped=0 pending=8188 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:692s;cuda2:trace_context:trace_bank__Self__llama3_8b__wikics__s3407:485s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:07:25 (UTC+8)
3006tasks done=3951/12141 running=2 failed=0 skipped=0 pending=8188 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:511s;cuda2:trace_context:trace_bank__Self__llama3_8b__wikics__s3407:304s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260821-01:04:24 (UTC+8)
3005tasks done=3951/12141 running=2 failed=0 skipped=0 pending=8188 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:329s;cuda2:trace_context:trace_bank__Self__llama3_8b__wikics__s3407:122s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260821-01:01:22 (UTC+8)
3004tasks done=3950/12141 running=2 failed=0 skipped=0 pending=8189 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s999:148s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:7557s | GPU cuda1:7851/24564MiB:0%;cuda2:14063/24564MiB:100%260821-00:58:21 (UTC+8)
3003tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:2322s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:7376s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:55:20 (UTC+8)
3002tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:2141s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:7195s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:52:19 (UTC+8)
3001tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:1959s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:7013s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:49:18 (UTC+8)
3000tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:1777s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:6831s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:46:15 (UTC+8)
2999tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:1596s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:6650s | GPU cuda1:7851/24564MiB:96%;cuda2:14063/24564MiB:100%260821-00:43:14 (UTC+8)
2998tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:1415s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:6469s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:40:13 (UTC+8)
2997tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:1234s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:6288s | GPU cuda1:7851/24564MiB:96%;cuda2:14063/24564MiB:100%260821-00:37:12 (UTC+8)
2996tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:1052s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:6106s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:34:11 (UTC+8)
2995tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:871s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:5925s | GPU cuda1:7851/24564MiB:96%;cuda2:14063/24564MiB:100%260821-00:31:09 (UTC+8)
2994tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:690s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:5744s | GPU cuda1:7851/24564MiB:96%;cuda2:14063/24564MiB:100%260821-00:28:08 (UTC+8)
2993tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:509s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:5563s | GPU cuda1:7851/24564MiB:94%;cuda2:14063/24564MiB:100%260821-00:25:07 (UTC+8)
2992tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:327s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:5381s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:22:05 (UTC+8)
2991tasks done=3948/12141 running=2 failed=0 skipped=0 pending=8191 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s999:145s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:5199s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:19:04 (UTC+8)
2990tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:2309s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:5018s | GPU cuda1:7851/24564MiB:96%;cuda2:14063/24564MiB:100%260821-00:16:02 (UTC+8)
2989tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:2127s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:4836s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:13:00 (UTC+8)
2988tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:1946s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:4655s | GPU cuda1:7851/24564MiB:95%;cuda2:14063/24564MiB:100%260821-00:09:58 (UTC+8)
2987tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:1765s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:4474s | GPU cuda1:7851/24564MiB:96%;cuda2:14063/24564MiB:100%260821-00:06:57 (UTC+8)
2986tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:1583s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:4292s | GPU cuda1:7851/24564MiB:100%;cuda2:14063/24564MiB:100%260821-00:03:56 (UTC+8)
2985tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:1402s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:4111s | GPU cuda1:7851/24564MiB:97%;cuda2:11403/24564MiB:100%260821-00:00:55 (UTC+8)
2984tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:1220s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:3929s | GPU cuda1:7851/24564MiB:97%;cuda2:11403/24564MiB:100%260820-23:57:54 (UTC+8)
2983tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:1039s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:3748s | GPU cuda1:7851/24564MiB:100%;cuda2:11403/24564MiB:100%260820-23:54:51 (UTC+8)
2982tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:858s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:3567s | GPU cuda1:7851/24564MiB:100%;cuda2:11403/24564MiB:100%260820-23:51:50 (UTC+8)
2981tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:676s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:3385s | GPU cuda1:7851/24564MiB:100%;cuda2:11403/24564MiB:100%260820-23:48:49 (UTC+8)
2980tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:495s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:3204s | GPU cuda1:7851/24564MiB:97%;cuda2:11403/24564MiB:100%260820-23:45:48 (UTC+8)
2979tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:314s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:3023s | GPU cuda1:7851/24564MiB:100%;cuda2:11403/24564MiB:100%260820-23:42:46 (UTC+8)
2978tasks done=3947/12141 running=2 failed=0 skipped=0 pending=8192 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s999:132s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:2841s | GPU cuda1:7851/24564MiB:96%;cuda2:11403/24564MiB:100%260820-23:39:45 (UTC+8)
2977tasks done=3945/12141 running=2 failed=0 skipped=0 pending=8194 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s999:1432s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:2660s | GPU cuda1:7851/24564MiB:97%;cuda2:11403/24564MiB:100%260820-23:36:43 (UTC+8)
2976tasks done=3945/12141 running=2 failed=0 skipped=0 pending=8194 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s999:1249s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:2477s | GPU cuda1:7851/24564MiB:98%;cuda2:11403/24564MiB:100%260820-23:33:42 (UTC+8)
2975tasks done=3945/12141 running=2 failed=0 skipped=0 pending=8194 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s999:1068s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:2296s | GPU cuda1:7851/24564MiB:98%;cuda2:11403/24564MiB:100%260820-23:30:40 (UTC+8)
2974tasks done=3945/12141 running=2 failed=0 skipped=0 pending=8194 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s999:886s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:2115s | GPU cuda1:7851/24564MiB:97%;cuda2:11403/24564MiB:100%260820-23:27:39 (UTC+8)
2973tasks done=3945/12141 running=2 failed=0 skipped=0 pending=8194 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s999:705s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:1933s | GPU cuda1:7851/24564MiB:98%;cuda2:11403/24564MiB:100%260820-23:24:37 (UTC+8)
2972tasks done=3945/12141 running=2 failed=0 skipped=0 pending=8194 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s999:524s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:1752s | GPU cuda1:7851/24564MiB:98%;cuda2:11403/24564MiB:100%260820-23:21:36 (UTC+8)
2971tasks done=3945/12141 running=2 failed=0 skipped=0 pending=8194 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s999:342s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:1571s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260820-23:18:35 (UTC+8)
2970tasks done=3945/12141 running=2 failed=0 skipped=0 pending=8194 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s999:161s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:1389s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260820-23:15:33 (UTC+8)
2969tasks done=3944/12141 running=2 failed=0 skipped=0 pending=8195 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s999:1462s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:1208s | GPU cuda1:7851/24564MiB:97%;cuda2:11257/24564MiB:100%260820-23:12:32 (UTC+8)
2968tasks done=3944/12141 running=2 failed=0 skipped=0 pending=8195 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s999:1279s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:1026s | GPU cuda1:7851/24564MiB:98%;cuda2:11257/24564MiB:100%260820-23:09:31 (UTC+8)
2967tasks done=3944/12141 running=2 failed=0 skipped=0 pending=8195 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s999:1097s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:844s | GPU cuda1:7851/24564MiB:98%;cuda2:11257/24564MiB:100%260820-23:06:28 (UTC+8)
2966tasks done=3944/12141 running=2 failed=0 skipped=0 pending=8195 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s999:916s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:663s | GPU cuda1:7851/24564MiB:98%;cuda2:11257/24564MiB:100%260820-23:03:27 (UTC+8)
2965tasks done=3944/12141 running=2 failed=0 skipped=0 pending=8195 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s999:735s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:482s | GPU cuda1:7851/24564MiB:98%;cuda2:14061/24564MiB:100%260820-23:00:25 (UTC+8)
2964tasks done=3944/12141 running=2 failed=0 skipped=0 pending=8195 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s999:554s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:300s | GPU cuda1:7851/24564MiB:97%;cuda2:14061/24564MiB:100%260820-22:57:24 (UTC+8)
2963tasks done=3944/12141 running=2 failed=0 skipped=0 pending=8195 | results done=2600/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s999:373s;cuda2:sft:sft__trace__llama3_8b__wikics__s3407:119s | GPU cuda1:7851/24564MiB:100%;cuda2:14061/24564MiB:100%260820-22:54:23 (UTC+8)
2962tasks done=3943/12141 running=2 failed=0 skipped=0 pending=8196 | results done=2599/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s999:191s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s888__gat3:279s | GPU cuda1:7851/24564MiB:100%;cuda2:16693/24564MiB:100%260820-22:51:22 (UTC+8)
2961tasks done=3943/12141 running=2 failed=0 skipped=0 pending=8196 | results done=2599/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s999:10s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s888__gat3:98s | GPU cuda1:7649/24564MiB:24%;cuda2:16693/24564MiB:100%260820-22:48:21 (UTC+8)
2960tasks done=3940/12141 running=2 failed=0 skipped=0 pending=8199 | results done=2598/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s999:733s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s888__gat3:257s | GPU cuda1:7851/24564MiB:97%;cuda2:16693/24564MiB:100%260820-22:45:19 (UTC+8)
2959tasks done=3940/12141 running=2 failed=0 skipped=0 pending=8199 | results done=2598/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s999:552s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s888__gat3:76s | GPU cuda1:7851/24564MiB:98%;cuda2:16691/24564MiB:100%260820-22:42:17 (UTC+8)
2958tasks done=3939/12141 running=2 failed=0 skipped=0 pending=8200 | results done=2597/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s999:370s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s888__gat3:236s | GPU cuda1:7851/24564MiB:98%;cuda2:16691/24564MiB:100%260820-22:39:16 (UTC+8)
2957tasks done=3939/12141 running=2 failed=0 skipped=0 pending=8200 | results done=2597/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s999:189s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s888__gat3:55s | GPU cuda1:7851/24564MiB:97%;cuda2:16691/24564MiB:100%260820-22:36:15 (UTC+8)
2956tasks done=3938/12141 running=2 failed=0 skipped=0 pending=8201 | results done=2596/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s999:8s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s888__gat3:215s | GPU cuda1:7637/24564MiB:25%;cuda2:16693/24564MiB:100%260820-22:33:14 (UTC+8)
2955tasks done=3937/12141 running=2 failed=0 skipped=0 pending=8202 | results done=2596/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s999:729s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s888__gat3:34s | GPU cuda1:7851/24564MiB:96%;cuda2:16691/24564MiB:100%260820-22:30:13 (UTC+8)
2954tasks done=3936/12141 running=2 failed=0 skipped=0 pending=8203 | results done=2595/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s999:547s;cuda2:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s888__gat3:185s | GPU cuda1:7851/24564MiB:97%;cuda2:16693/24564MiB:100%260820-22:27:11 (UTC+8)
2953tasks done=3936/12141 running=2 failed=0 skipped=0 pending=8203 | results done=2595/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s999:366s;cuda2:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s888__gat3:4s | GPU cuda1:7851/24564MiB:97%;cuda2:11457/24564MiB:100%260820-22:24:10 (UTC+8)
2952tasks done=3935/12141 running=2 failed=0 skipped=0 pending=8204 | results done=2594/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s999:185s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s888__gat3:165s | GPU cuda1:7851/24564MiB:95%;cuda2:16693/24564MiB:100%260820-22:21:09 (UTC+8)
2951tasks done=3934/12141 running=2 failed=0 skipped=0 pending=8205 | results done=2593/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s999:2s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s888__gat3:324s | GPU cuda1:401/24564MiB:0%;cuda2:16693/24564MiB:100%260820-22:18:08 (UTC+8)
2950tasks done=3932/12141 running=2 failed=0 skipped=0 pending=8207 | results done=2593/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:2149s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s888__gat3:143s | GPU cuda1:7851/24564MiB:96%;cuda2:16693/24564MiB:100%260820-22:15:05 (UTC+8)
2949tasks done=3931/12141 running=2 failed=0 skipped=0 pending=8208 | results done=2592/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:1968s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s888__gat3:274s | GPU cuda1:7851/24564MiB:96%;cuda2:16693/24564MiB:100%260820-22:12:04 (UTC+8)
2948tasks done=3931/12141 running=2 failed=0 skipped=0 pending=8208 | results done=2592/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:1787s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s888__gat3:93s | GPU cuda1:7851/24564MiB:96%;cuda2:16693/24564MiB:100%260820-22:09:03 (UTC+8)
2947tasks done=3930/12141 running=2 failed=0 skipped=0 pending=8209 | results done=2591/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:1606s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s888__gat3:253s | GPU cuda1:7851/24564MiB:96%;cuda2:16693/24564MiB:100%260820-22:06:02 (UTC+8)
2946tasks done=3930/12141 running=2 failed=0 skipped=0 pending=8209 | results done=2591/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:1424s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s888__gat3:70s | GPU cuda1:7851/24564MiB:96%;cuda2:16691/24564MiB:100%260820-22:03:01 (UTC+8)
2945tasks done=3929/12141 running=2 failed=0 skipped=0 pending=8210 | results done=2590/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:1243s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s888__gat3:230s | GPU cuda1:7851/24564MiB:100%;cuda2:16691/24564MiB:100%260820-21:59:59 (UTC+8)
2944tasks done=3929/12141 running=2 failed=0 skipped=0 pending=8210 | results done=2590/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:1062s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s888__gat3:49s | GPU cuda1:7851/24564MiB:100%;cuda2:16691/24564MiB:100%260820-21:56:57 (UTC+8)
2943tasks done=3928/12141 running=2 failed=0 skipped=0 pending=8211 | results done=2589/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:881s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s888__gat2:112s | GPU cuda1:7851/24564MiB:96%;cuda2:13145/24564MiB:100%260820-21:53:56 (UTC+8)
2942tasks done=3927/12141 running=2 failed=0 skipped=0 pending=8212 | results done=2588/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:698s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s888__gat2:172s | GPU cuda1:7851/24564MiB:96%;cuda2:13145/24564MiB:100%260820-21:50:55 (UTC+8)
2941tasks done=3926/12141 running=2 failed=0 skipped=0 pending=8213 | results done=2587/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:517s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s888__gat2:235s | GPU cuda1:7851/24564MiB:96%;cuda2:13145/24564MiB:100%260820-21:47:52 (UTC+8)
2940tasks done=3926/12141 running=2 failed=0 skipped=0 pending=8213 | results done=2587/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:335s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s888__gat2:53s | GPU cuda1:7851/24564MiB:100%;cuda2:13145/24564MiB:100%260820-21:44:51 (UTC+8)
2939tasks done=3925/12141 running=2 failed=0 skipped=0 pending=8214 | results done=2586/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s999:153s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s888__gat2:115s | GPU cuda1:7851/24564MiB:100%;cuda2:13145/24564MiB:100%260820-21:41:49 (UTC+8)
2938tasks done=3923/12141 running=2 failed=0 skipped=0 pending=8216 | results done=2585/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:2298s;cuda2:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s888__gat2:178s | GPU cuda1:7851/24564MiB:95%;cuda2:13145/24564MiB:100%260820-21:38:48 (UTC+8)
2937tasks done=3922/12141 running=2 failed=0 skipped=0 pending=8217 | results done=2584/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:2117s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s888__gat2:240s | GPU cuda1:7851/24564MiB:95%;cuda2:13145/24564MiB:100%260820-21:35:47 (UTC+8)
2936tasks done=3922/12141 running=2 failed=0 skipped=0 pending=8217 | results done=2584/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:1936s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s888__gat2:59s | GPU cuda1:7851/24564MiB:100%;cuda2:13145/24564MiB:100%260820-21:32:46 (UTC+8)
2935tasks done=3921/12141 running=2 failed=0 skipped=0 pending=8218 | results done=2583/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:1754s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s888__gat2:121s | GPU cuda1:7851/24564MiB:100%;cuda2:13145/24564MiB:100%260820-21:29:45 (UTC+8)
2934tasks done=3920/12141 running=2 failed=0 skipped=0 pending=8219 | results done=2582/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:1573s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s888__gat2:181s | GPU cuda1:7851/24564MiB:96%;cuda2:13145/24564MiB:100%260820-21:26:43 (UTC+8)
2933tasks done=3920/12141 running=2 failed=0 skipped=0 pending=8219 | results done=2582/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:1392s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s888__gat2:0s | GPU cuda1:7851/24564MiB:96%;cuda2:3597/24564MiB:100%260820-21:23:42 (UTC+8)
2932tasks done=3919/12141 running=2 failed=0 skipped=0 pending=8220 | results done=2581/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:1211s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s888__gat2:60s | GPU cuda1:7851/24564MiB:100%;cuda2:13145/24564MiB:100%260820-21:20:41 (UTC+8)
2931tasks done=3918/12141 running=2 failed=0 skipped=0 pending=8221 | results done=2580/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:1028s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s888__gat2:119s | GPU cuda1:7851/24564MiB:96%;cuda2:13145/24564MiB:100%260820-21:17:40 (UTC+8)
2930tasks done=3917/12141 running=2 failed=0 skipped=0 pending=8222 | results done=2579/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:847s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s888__gat1:80s | GPU cuda1:7851/24564MiB:97%;cuda2:11277/24564MiB:100%260820-21:14:37 (UTC+8)
2929tasks done=3915/12141 running=2 failed=0 skipped=0 pending=8224 | results done=2577/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:665s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s888__gat1:66s | GPU cuda1:7851/24564MiB:97%;cuda2:11277/24564MiB:100%260820-21:11:36 (UTC+8)
2928tasks done=3913/12141 running=2 failed=0 skipped=0 pending=8226 | results done=2575/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:484s;cuda2:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s888__gat1:43s | GPU cuda1:7851/24564MiB:100%;cuda2:11277/24564MiB:100%260820-21:08:34 (UTC+8)
2927tasks done=3912/12141 running=2 failed=0 skipped=0 pending=8227 | results done=2574/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:303s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s888__gat1:3s | GPU cuda1:7851/24564MiB:100%;cuda2:4221/24564MiB:100%260820-21:05:33 (UTC+8)
2926tasks done=3910/12141 running=2 failed=0 skipped=0 pending=8229 | results done=2572/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s999:122s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s888__gat1:14s | GPU cuda1:7851/24564MiB:100%;cuda2:11277/24564MiB:100%260820-21:02:32 (UTC+8)
2925tasks done=3905/12141 running=2 failed=0 skipped=0 pending=8234 | results done=2569/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s999:1401s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s888__sage3:34s | GPU cuda1:7851/24564MiB:97%;cuda2:5025/24564MiB:100%260820-20:59:31 (UTC+8)
2924tasks done=3903/12141 running=2 failed=0 skipped=0 pending=8236 | results done=2567/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s999:1219s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s888__sage3:33s | GPU cuda1:7851/24564MiB:97%;cuda2:5023/24564MiB:100%260820-20:56:29 (UTC+8)
2923tasks done=3901/12141 running=2 failed=0 skipped=0 pending=8238 | results done=2565/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s999:1038s;cuda2:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s888__sage3:42s | GPU cuda1:7851/24564MiB:97%;cuda2:5025/24564MiB:100%260820-20:53:28 (UTC+8)
2922tasks done=3899/12141 running=2 failed=0 skipped=0 pending=8240 | results done=2563/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s999:857s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s888__sage3:18s | GPU cuda1:7851/24564MiB:97%;cuda2:5023/24564MiB:100%260820-20:50:27 (UTC+8)
2921tasks done=3897/12141 running=2 failed=0 skipped=0 pending=8242 | results done=2561/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s999:676s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s888__sage3:1s | GPU cuda1:7851/24564MiB:97%;cuda2:3597/24564MiB:100%260820-20:47:26 (UTC+8)
2920tasks done=3894/12141 running=2 failed=0 skipped=0 pending=8245 | results done=2558/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s999:494s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s888__sage2:74s | GPU cuda1:7851/24564MiB:97%;cuda2:5019/24564MiB:100%260820-20:44:24 (UTC+8)
2919tasks done=3892/12141 running=2 failed=0 skipped=0 pending=8247 | results done=2556/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s999:313s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s888__sage2:64s | GPU cuda1:7851/24564MiB:97%;cuda2:5019/24564MiB:100%260820-20:41:23 (UTC+8)
2918tasks done=3890/12141 running=2 failed=0 skipped=0 pending=8249 | results done=2554/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s999:132s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s888__sage2:45s | GPU cuda1:7851/24564MiB:97%;cuda2:5019/24564MiB:100%260820-20:38:22 (UTC+8)
2917tasks done=3887/12141 running=2 failed=0 skipped=0 pending=8252 | results done=2552/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s999:1409s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s888__sage2:4s | GPU cuda1:7851/24564MiB:97%;cuda2:5005/24564MiB:100%260820-20:35:21 (UTC+8)
2916tasks done=3884/12141 running=2 failed=0 skipped=0 pending=8255 | results done=2549/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s999:1226s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s888__sage1:4s | GPU cuda1:7851/24564MiB:97%;cuda2:4635/24564MiB:100%260820-20:32:20 (UTC+8)
2915tasks done=3881/12141 running=2 failed=0 skipped=0 pending=8258 | results done=2546/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s999:1044s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s888__sage1:3s | GPU cuda1:7851/24564MiB:97%;cuda2:4561/24564MiB:100%260820-20:29:16 (UTC+8)
2914tasks done=3876/12141 running=2 failed=0 skipped=0 pending=8263 | results done=2541/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s999:864s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s888__sage1:8s | GPU cuda1:7851/24564MiB:100%;cuda2:4635/24564MiB:100%260820-20:26:15 (UTC+8)
2913tasks done=3873/12141 running=2 failed=0 skipped=0 pending=8266 | results done=2538/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s999:682s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s888__gcn3:113s | GPU cuda1:7851/24564MiB:97%;cuda2:5221/24564MiB:100%260820-20:23:14 (UTC+8)
2912tasks done=3872/12141 running=2 failed=0 skipped=0 pending=8267 | results done=2537/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s999:500s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s888__gcn3:60s | GPU cuda1:7851/24564MiB:97%;cuda2:5219/24564MiB:100%260820-20:20:13 (UTC+8)
2911tasks done=3871/12141 running=2 failed=0 skipped=0 pending=8268 | results done=2536/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s999:317s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s888__gcn3:8s | GPU cuda1:7851/24564MiB:97%;cuda2:5219/24564MiB:100%260820-20:17:11 (UTC+8)
2910tasks done=3869/12141 running=2 failed=0 skipped=0 pending=8270 | results done=2534/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s999:136s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s888__gcn3:87s | GPU cuda1:7851/24564MiB:97%;cuda2:5221/24564MiB:100%260820-20:14:08 (UTC+8)
2909tasks done=3866/12141 running=2 failed=0 skipped=0 pending=8273 | results done=2533/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s999:1013s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s888__gcn3:36s | GPU cuda1:7851/24564MiB:97%;cuda2:5221/24564MiB:100%260820-20:11:07 (UTC+8)
2908tasks done=3864/12141 running=2 failed=0 skipped=0 pending=8275 | results done=2531/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s999:832s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s888__gcn3:115s | GPU cuda1:7851/24564MiB:97%;cuda2:5221/24564MiB:100%260820-20:08:06 (UTC+8)
2907tasks done=3863/12141 running=2 failed=0 skipped=0 pending=8276 | results done=2530/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s999:651s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s888__gcn3:63s | GPU cuda1:7851/24564MiB:97%;cuda2:5219/24564MiB:100%260820-20:05:05 (UTC+8)
2906tasks done=3860/12141 running=2 failed=0 skipped=0 pending=8279 | results done=2527/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s999:469s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s888__gcn1:11s | GPU cuda1:7851/24564MiB:97%;cuda2:5093/24564MiB:100%260820-20:02:04 (UTC+8)
2905tasks done=3857/12141 running=2 failed=0 skipped=0 pending=8282 | results done=2524/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s999:288s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s888__gcn1:16s | GPU cuda1:7851/24564MiB:98%;cuda2:5093/24564MiB:100%260820-19:59:02 (UTC+8)
2904tasks done=3853/12141 running=2 failed=0 skipped=0 pending=8286 | results done=2520/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s999:107s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s888__gcn1:17s | GPU cuda1:7851/24564MiB:0%;cuda2:5093/24564MiB:100%260820-19:56:01 (UTC+8)
2903tasks done=3850/12141 running=2 failed=0 skipped=0 pending=8289 | results done=2518/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s999:982s;cuda2:hcr_downstream:run__E4_ReliabilityFeature__llama3_8b__wikics__s888__gcn2:39s | GPU cuda1:7851/24564MiB:95%;cuda2:5137/24564MiB:100%260820-19:53:00 (UTC+8)
2902tasks done=3848/12141 running=2 failed=0 skipped=0 pending=8291 | results done=2516/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s999:801s;cuda2:hcr_downstream:run__E2_NoContextMetadata__llama3_8b__wikics__s888__gcn2:54s | GPU cuda1:7851/24564MiB:97%;cuda2:5137/24564MiB:100%260820-19:49:59 (UTC+8)
2901tasks done=3846/12141 running=2 failed=0 skipped=0 pending=8293 | results done=2514/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s999:619s;cuda2:hcr_downstream:run__E1_Snapshot1__llama3_8b__wikics__s888__gcn2:76s | GPU cuda1:7851/24564MiB:95%;cuda2:5137/24564MiB:100%260820-19:46:58 (UTC+8)
2900tasks done=3844/12141 running=2 failed=0 skipped=0 pending=8295 | results done=2512/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s999:438s;cuda2:hcr_downstream:run__J_Labels25__llama3_8b__wikics__s888__gcn2:98s | GPU cuda1:7851/24564MiB:97%;cuda2:5137/24564MiB:100%260820-19:43:57 (UTC+8)
2899tasks done=3843/12141 running=2 failed=0 skipped=0 pending=8296 | results done=2511/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s999:257s;cuda2:hcr_downstream:run__J_Labels10__llama3_8b__wikics__s888__gcn2:12s | GPU cuda1:7851/24564MiB:97%;cuda2:5137/24564MiB:100%260820-19:40:56 (UTC+8)
2898tasks done=3841/12141 running=2 failed=0 skipped=0 pending=8298 | results done=2509/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s999:76s;cuda2:hcr_downstream:run__G_ProbeT1__llama3_8b__wikics__s888__gcn2:31s | GPU cuda1:7851/24564MiB:97%;cuda2:5137/24564MiB:100%260820-19:37:55 (UTC+8)
2897tasks done=3837/12141 running=2 failed=0 skipped=0 pending=8302 | results done=2507/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s999:773s;cuda2:hcr_downstream:run__F_EtaQuarter__llama3_8b__wikics__s888__gcn2:47s | GPU cuda1:7851/24564MiB:95%;cuda2:5137/24564MiB:100%260820-19:34:54 (UTC+8)
2896tasks done=3835/12141 running=2 failed=0 skipped=0 pending=8304 | results done=2505/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s999:591s;cuda2:hcr_downstream:run__D_TeacherTarget__llama3_8b__wikics__s888__gcn2:66s | GPU cuda1:7851/24564MiB:97%;cuda2:5137/24564MiB:100%260820-19:31:52 (UTC+8)
2895tasks done=3833/12141 running=2 failed=0 skipped=0 pending=8306 | results done=2503/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s999:410s;cuda2:hcr_downstream:run__B_RemoveLocal__llama3_8b__wikics__s888__gcn2:88s | GPU cuda1:7851/24564MiB:95%;cuda2:5137/24564MiB:100%260820-19:28:50 (UTC+8)
2894tasks done=3832/12141 running=2 failed=0 skipped=0 pending=8307 | results done=2502/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s999:229s;cuda2:hcr_downstream:run__A_RemoveGlobal__llama3_8b__wikics__s888__gcn2:9s | GPU cuda1:7851/24564MiB:95%;cuda2:5137/24564MiB:100%260820-19:25:49 (UTC+8)
2893tasks done=3830/12141 running=2 failed=0 skipped=0 pending=8309 | results done=2500/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s999:48s;cuda2:hcr_downstream:run__HCR_EB_NoRegion__llama3_8b__wikics__s888__gcn2:30s | GPU cuda1:7851/24564MiB:98%;cuda2:5137/24564MiB:100%260820-19:22:48 (UTC+8)
2892tasks done=3827/12141 running=2 failed=0 skipped=0 pending=8312 | results done=2498/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s999:742s;cuda2:hcr_downstream:run__HCR_EB_Teacher__llama3_8b__wikics__s888__gcn2:51s | GPU cuda1:7851/24564MiB:95%;cuda2:5137/24564MiB:100%260820-19:19:47 (UTC+8)
2891tasks done=3825/12141 running=2 failed=0 skipped=0 pending=8314 | results done=2496/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s999:561s;cuda2:hcr_downstream:run__GlobalOnly__llama3_8b__wikics__s888__gcn2:73s | GPU cuda1:7851/24564MiB:98%;cuda2:5137/24564MiB:100%260820-19:16:45 (UTC+8)
2890tasks done=3823/12141 running=2 failed=0 skipped=0 pending=8316 | results done=2494/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s999:380s;cuda2:hcr_downstream:run__HCR_Arch__llama3_8b__wikics__s888__gcn2:95s | GPU cuda1:7851/24564MiB:97%;cuda2:5137/24564MiB:100%260820-19:13:44 (UTC+8)
2889tasks done=3822/12141 running=2 failed=0 skipped=0 pending=8317 | results done=2493/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s999:199s;cuda2:hcr_downstream:run__HCR_Teacher__llama3_8b__wikics__s888__gcn2:15s | GPU cuda1:7851/24564MiB:95%;cuda2:5137/24564MiB:100%260820-19:10:43 (UTC+8)
2888tasks done=3820/12141 running=2 failed=0 skipped=0 pending=8319 | results done=2491/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s999:18s;cuda2:rci_downstream:run__RCI_Static__llama3_8b__wikics__s888__gcn2:232s | GPU cuda1:7701/24564MiB:24%;cuda2:5139/24564MiB:100%260820-19:07:42 (UTC+8)
2887tasks done=3818/12141 running=2 failed=0 skipped=0 pending=8321 | results done=2491/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:2164s;cuda2:rci_downstream:run__RCI_Static__llama3_8b__wikics__s888__gcn2:51s | GPU cuda1:7851/24564MiB:95%;cuda2:5133/24564MiB:100%260820-19:04:41 (UTC+8)
2886tasks done=3816/12141 running=2 failed=0 skipped=0 pending=8323 | results done=2489/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:1982s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s888__gcn2:72s | GPU cuda1:7851/24564MiB:96%;cuda2:5137/24564MiB:100%260820-19:01:39 (UTC+8)
2885tasks done=3814/12141 running=2 failed=0 skipped=0 pending=8325 | results done=2487/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:1801s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s888__gcn2:93s | GPU cuda1:7851/24564MiB:96%;cuda2:5137/24564MiB:100%260820-18:58:38 (UTC+8)
2884tasks done=3812/12141 running=2 failed=0 skipped=0 pending=8327 | results done=2485/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:1620s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s888__gcn2:94s | GPU cuda1:7851/24564MiB:100%;cuda2:5137/24564MiB:100%260820-18:55:36 (UTC+8)
2883tasks done=3811/12141 running=2 failed=0 skipped=0 pending=8328 | results done=2484/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:1439s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s888__gcn2:14s | GPU cuda1:7851/24564MiB:96%;cuda2:5137/24564MiB:100%260820-18:52:35 (UTC+8)
2882tasks done=3809/12141 running=2 failed=0 skipped=0 pending=8330 | results done=2482/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:1257s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s888__gcn2:35s | GPU cuda1:7851/24564MiB:100%;cuda2:5137/24564MiB:100%260820-18:49:34 (UTC+8)
2881tasks done=3807/12141 running=2 failed=0 skipped=0 pending=8332 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:1076s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:18669s | GPU cuda1:7851/24564MiB:97%;cuda2:4107/24564MiB:100%260820-18:46:33 (UTC+8)
2880tasks done=3807/12141 running=2 failed=0 skipped=0 pending=8332 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:895s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:18488s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-18:43:31 (UTC+8)
2879tasks done=3807/12141 running=2 failed=0 skipped=0 pending=8332 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:714s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:18306s | GPU cuda1:7851/24564MiB:100%;cuda2:4109/24564MiB:100%260820-18:40:30 (UTC+8)
2878tasks done=3807/12141 running=2 failed=0 skipped=0 pending=8332 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:532s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:18125s | GPU cuda1:7851/24564MiB:100%;cuda2:4109/24564MiB:100%260820-18:37:29 (UTC+8)
2877tasks done=3807/12141 running=2 failed=0 skipped=0 pending=8332 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:351s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:17943s | GPU cuda1:7851/24564MiB:100%;cuda2:4107/24564MiB:100%260820-18:34:28 (UTC+8)
2876tasks done=3807/12141 running=2 failed=0 skipped=0 pending=8332 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s999:169s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:17762s | GPU cuda1:7851/24564MiB:100%;cuda2:4109/24564MiB:100%260820-18:31:26 (UTC+8)
2875tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:2311s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:17581s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-18:28:25 (UTC+8)
2874tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:2130s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:17400s | GPU cuda1:7851/24564MiB:96%;cuda2:4109/24564MiB:100%260820-18:25:24 (UTC+8)
2873tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:1948s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:17218s | GPU cuda1:7851/24564MiB:93%;cuda2:4107/24564MiB:100%260820-18:22:23 (UTC+8)
2872tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:1767s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:17037s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-18:19:21 (UTC+8)
2871tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:1586s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:16856s | GPU cuda1:7851/24564MiB:100%;cuda2:4109/24564MiB:100%260820-18:16:20 (UTC+8)
2870tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:1405s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:16675s | GPU cuda1:7851/24564MiB:100%;cuda2:4109/24564MiB:100%260820-18:13:18 (UTC+8)
2869tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:1223s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:16493s | GPU cuda1:7851/24564MiB:100%;cuda2:4109/24564MiB:100%260820-18:10:17 (UTC+8)
2868tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:1042s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:16312s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-18:07:16 (UTC+8)
2867tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:861s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:16131s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-18:04:14 (UTC+8)
2866tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:679s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:15949s | GPU cuda1:7851/24564MiB:100%;cuda2:4107/24564MiB:100%260820-18:01:13 (UTC+8)
2865tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:498s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:15768s | GPU cuda1:7851/24564MiB:100%;cuda2:4109/24564MiB:100%260820-17:58:12 (UTC+8)
2864tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:316s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:15586s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-17:55:11 (UTC+8)
2863tasks done=3806/12141 running=2 failed=0 skipped=0 pending=8333 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s999:135s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:15405s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-17:52:09 (UTC+8)
2862tasks done=3804/12141 running=2 failed=0 skipped=0 pending=8335 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s999:1013s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:15224s | GPU cuda1:7851/24564MiB:97%;cuda2:4107/24564MiB:100%260820-17:49:08 (UTC+8)
2861tasks done=3804/12141 running=2 failed=0 skipped=0 pending=8335 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s999:831s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:15042s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-17:46:07 (UTC+8)
2860tasks done=3804/12141 running=2 failed=0 skipped=0 pending=8335 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s999:650s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:14861s | GPU cuda1:7851/24564MiB:95%;cuda2:4109/24564MiB:100%260820-17:43:04 (UTC+8)
2859tasks done=3804/12141 running=2 failed=0 skipped=0 pending=8335 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s999:468s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:14680s | GPU cuda1:7851/24564MiB:94%;cuda2:4109/24564MiB:100%260820-17:40:03 (UTC+8)
2858tasks done=3804/12141 running=2 failed=0 skipped=0 pending=8335 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s999:287s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:14498s | GPU cuda1:7851/24564MiB:97%;cuda2:4107/24564MiB:100%260820-17:37:02 (UTC+8)
2857tasks done=3804/12141 running=2 failed=0 skipped=0 pending=8335 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s999:106s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:14317s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-17:34:01 (UTC+8)
2856tasks done=3803/12141 running=2 failed=0 skipped=0 pending=8336 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s999:981s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:14135s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-17:30:59 (UTC+8)
2855tasks done=3803/12141 running=2 failed=0 skipped=0 pending=8336 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s999:800s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:13954s | GPU cuda1:7851/24564MiB:96%;cuda2:4109/24564MiB:100%260820-17:27:58 (UTC+8)
2854tasks done=3803/12141 running=2 failed=0 skipped=0 pending=8336 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s999:619s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:13773s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-17:24:57 (UTC+8)
2853tasks done=3803/12141 running=2 failed=0 skipped=0 pending=8336 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s999:437s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:13591s | GPU cuda1:7851/24564MiB:96%;cuda2:4109/24564MiB:100%260820-17:21:56 (UTC+8)
2852tasks done=3803/12141 running=2 failed=0 skipped=0 pending=8336 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s999:254s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:13408s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-17:18:54 (UTC+8)
2851tasks done=3803/12141 running=2 failed=0 skipped=0 pending=8336 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s999:73s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:13226s | GPU cuda1:7851/24564MiB:98%;cuda2:4107/24564MiB:100%260820-17:15:51 (UTC+8)
2850tasks done=3801/12141 running=2 failed=0 skipped=0 pending=8338 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s999:774s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:13045s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-17:12:49 (UTC+8)
2849tasks done=3801/12141 running=2 failed=0 skipped=0 pending=8338 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s999:593s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:12864s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-17:09:48 (UTC+8)
2848tasks done=3801/12141 running=2 failed=0 skipped=0 pending=8338 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s999:412s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:12683s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-17:06:47 (UTC+8)
2847tasks done=3801/12141 running=2 failed=0 skipped=0 pending=8338 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s999:230s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:12501s | GPU cuda1:7851/24564MiB:93%;cuda2:4107/24564MiB:100%260820-17:03:46 (UTC+8)
2846tasks done=3801/12141 running=2 failed=0 skipped=0 pending=8338 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s999:45s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:12316s | GPU cuda1:7851/24564MiB:94%;cuda2:4109/24564MiB:100%260820-17:00:44 (UTC+8)
2845tasks done=3800/12141 running=2 failed=0 skipped=0 pending=8339 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s999:745s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:12135s | GPU cuda1:7851/24564MiB:95%;cuda2:4109/24564MiB:100%260820-16:57:39 (UTC+8)
2844tasks done=3800/12141 running=2 failed=0 skipped=0 pending=8339 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s999:564s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:11954s | GPU cuda1:7851/24564MiB:93%;cuda2:4109/24564MiB:100%260820-16:54:38 (UTC+8)
2843tasks done=3800/12141 running=2 failed=0 skipped=0 pending=8339 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s999:383s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:11773s | GPU cuda1:7851/24564MiB:96%;cuda2:4107/24564MiB:100%260820-16:51:37 (UTC+8)
2842tasks done=3800/12141 running=2 failed=0 skipped=0 pending=8339 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s999:201s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:11592s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-16:48:36 (UTC+8)
2841tasks done=3800/12141 running=2 failed=0 skipped=0 pending=8339 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s999:20s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:11411s | GPU cuda1:7715/24564MiB:21%;cuda2:4109/24564MiB:100%260820-16:45:34 (UTC+8)
2840tasks done=3798/12141 running=2 failed=0 skipped=0 pending=8341 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s999:930s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:11229s | GPU cuda1:7851/24564MiB:96%;cuda2:4109/24564MiB:100%260820-16:42:33 (UTC+8)
2839tasks done=3798/12141 running=2 failed=0 skipped=0 pending=8341 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s999:748s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:11048s | GPU cuda1:7851/24564MiB:95%;cuda2:4107/24564MiB:100%260820-16:39:32 (UTC+8)
2838tasks done=3798/12141 running=2 failed=0 skipped=0 pending=8341 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s999:567s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:10867s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-16:36:30 (UTC+8)
2837tasks done=3798/12141 running=2 failed=0 skipped=0 pending=8341 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s999:386s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:10685s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-16:33:29 (UTC+8)
2836tasks done=3798/12141 running=2 failed=0 skipped=0 pending=8341 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s999:205s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:10504s | GPU cuda1:7851/24564MiB:0%;cuda2:4109/24564MiB:100%260820-16:30:28 (UTC+8)
2835tasks done=3798/12141 running=2 failed=0 skipped=0 pending=8341 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s999:23s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:10323s | GPU cuda1:7739/24564MiB:9%;cuda2:4109/24564MiB:100%260820-16:27:27 (UTC+8)
2834tasks done=3797/12141 running=2 failed=0 skipped=0 pending=8342 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s999:930s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:10141s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-16:24:26 (UTC+8)
2833tasks done=3797/12141 running=2 failed=0 skipped=0 pending=8342 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s999:749s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:9960s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-16:21:24 (UTC+8)
2832tasks done=3797/12141 running=2 failed=0 skipped=0 pending=8342 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s999:568s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:9779s | GPU cuda1:7851/24564MiB:98%;cuda2:4107/24564MiB:100%260820-16:18:23 (UTC+8)
2831tasks done=3797/12141 running=2 failed=0 skipped=0 pending=8342 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s999:387s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:9598s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-16:15:22 (UTC+8)
2830tasks done=3797/12141 running=2 failed=0 skipped=0 pending=8342 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s999:204s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:9415s | GPU cuda1:7851/24564MiB:96%;cuda2:4109/24564MiB:100%260820-16:12:20 (UTC+8)
2829tasks done=3797/12141 running=2 failed=0 skipped=0 pending=8342 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s999:21s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:9232s | GPU cuda1:7727/24564MiB:24%;cuda2:4109/24564MiB:100%260820-16:09:18 (UTC+8)
2828tasks done=3795/12141 running=2 failed=0 skipped=0 pending=8344 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s999:1304s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:9051s | GPU cuda1:7851/24564MiB:97%;cuda2:4107/24564MiB:100%260820-16:06:15 (UTC+8)
2827tasks done=3795/12141 running=2 failed=0 skipped=0 pending=8344 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s999:1123s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:8870s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-16:03:14 (UTC+8)
2826tasks done=3795/12141 running=2 failed=0 skipped=0 pending=8344 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s999:940s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:8688s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-16:00:13 (UTC+8)
2825tasks done=3795/12141 running=2 failed=0 skipped=0 pending=8344 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s999:759s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:8506s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-15:57:10 (UTC+8)
2824tasks done=3795/12141 running=2 failed=0 skipped=0 pending=8344 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s999:578s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:8325s | GPU cuda1:7851/24564MiB:97%;cuda2:4107/24564MiB:100%260820-15:54:09 (UTC+8)
2823tasks done=3795/12141 running=2 failed=0 skipped=0 pending=8344 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s999:397s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:8144s | GPU cuda1:7851/24564MiB:98%;cuda2:4109/24564MiB:100%260820-15:51:08 (UTC+8)
2822tasks done=3795/12141 running=2 failed=0 skipped=0 pending=8344 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s999:216s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:7963s | GPU cuda1:7851/24564MiB:93%;cuda2:4109/24564MiB:100%260820-15:48:07 (UTC+8)
2821tasks done=3795/12141 running=2 failed=0 skipped=0 pending=8344 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s999:35s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:7782s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:47%260820-15:45:06 (UTC+8)
2820tasks done=3794/12141 running=2 failed=0 skipped=0 pending=8345 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s999:1315s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:7601s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-15:42:05 (UTC+8)
2819tasks done=3794/12141 running=2 failed=0 skipped=0 pending=8345 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s999:1134s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:7420s | GPU cuda1:7851/24564MiB:100%;cuda2:4109/24564MiB:100%260820-15:39:04 (UTC+8)
2818tasks done=3794/12141 running=2 failed=0 skipped=0 pending=8345 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s999:953s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:7239s | GPU cuda1:7851/24564MiB:96%;cuda2:4109/24564MiB:100%260820-15:36:03 (UTC+8)
2817tasks done=3794/12141 running=2 failed=0 skipped=0 pending=8345 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s999:772s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:7058s | GPU cuda1:7851/24564MiB:94%;cuda2:4109/24564MiB:100%260820-15:33:02 (UTC+8)
2816tasks done=3794/12141 running=2 failed=0 skipped=0 pending=8345 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s999:590s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:6877s | GPU cuda1:7851/24564MiB:94%;cuda2:4109/24564MiB:100%260820-15:30:01 (UTC+8)
2815tasks done=3794/12141 running=2 failed=0 skipped=0 pending=8345 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s999:409s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:6695s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-15:26:59 (UTC+8)
2814tasks done=3794/12141 running=2 failed=0 skipped=0 pending=8345 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s999:228s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:6514s | GPU cuda1:7851/24564MiB:96%;cuda2:4107/24564MiB:100%260820-15:23:58 (UTC+8)
2813tasks done=3794/12141 running=2 failed=0 skipped=0 pending=8345 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s999:47s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:6333s | GPU cuda1:7851/24564MiB:97%;cuda2:4109/24564MiB:100%260820-15:20:57 (UTC+8)
2812tasks done=3793/12141 running=2 failed=0 skipped=0 pending=8346 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Self__llama3_8b__wikics__s999:602s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:6151s | GPU cuda1:7851/24564MiB:95%;cuda2:4109/24564MiB:100%260820-15:17:56 (UTC+8)
2811tasks done=3793/12141 running=2 failed=0 skipped=0 pending=8346 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Self__llama3_8b__wikics__s999:420s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:5970s | GPU cuda1:7851/24564MiB:96%;cuda2:4109/24564MiB:100%260820-15:14:54 (UTC+8)
2810tasks done=3793/12141 running=2 failed=0 skipped=0 pending=8346 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Self__llama3_8b__wikics__s999:239s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:5789s | GPU cuda1:7851/24564MiB:95%;cuda2:4107/24564MiB:100%260820-15:11:53 (UTC+8)
2809tasks done=3793/12141 running=2 failed=0 skipped=0 pending=8346 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Self__llama3_8b__wikics__s999:58s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:5608s | GPU cuda1:7851/24564MiB:96%;cuda2:4109/24564MiB:100%260820-15:08:52 (UTC+8)
2808tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:3820s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:5427s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-15:05:51 (UTC+8)
2807tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:3639s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:5246s | GPU cuda1:10477/24564MiB:99%;cuda2:4109/24564MiB:100%260820-15:02:50 (UTC+8)
2806tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:3458s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:5065s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:59:49 (UTC+8)
2805tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:3277s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:4883s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:56:48 (UTC+8)
2804tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:3096s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:4702s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:53:46 (UTC+8)
2803tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:2914s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:4521s | GPU cuda1:10477/24564MiB:99%;cuda2:4109/24564MiB:100%260820-14:50:45 (UTC+8)
2802tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:2733s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:4340s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:47:44 (UTC+8)
2801tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:2552s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:4159s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:44:43 (UTC+8)
2800tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:2371s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:3978s | GPU cuda1:10477/24564MiB:99%;cuda2:4109/24564MiB:100%260820-14:41:42 (UTC+8)
2799tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:2189s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:3796s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:38:41 (UTC+8)
2798tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:2008s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:3615s | GPU cuda1:10477/24564MiB:77%;cuda2:4109/24564MiB:100%260820-14:35:39 (UTC+8)
2797tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:1827s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:3434s | GPU cuda1:10477/24564MiB:99%;cuda2:4107/24564MiB:100%260820-14:32:38 (UTC+8)
2796tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:1646s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:3253s | GPU cuda1:10477/24564MiB:99%;cuda2:4109/24564MiB:100%260820-14:29:37 (UTC+8)
2795tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:1465s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:3072s | GPU cuda1:10477/24564MiB:99%;cuda2:4109/24564MiB:100%260820-14:26:36 (UTC+8)
2794tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:1283s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:2890s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:23:35 (UTC+8)
2793tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:1101s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:2708s | GPU cuda1:10477/24564MiB:100%;cuda2:4107/24564MiB:100%260820-14:20:33 (UTC+8)
2792tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:919s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:2526s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:17:30 (UTC+8)
2791tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:738s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:2345s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:14:29 (UTC+8)
2790tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:557s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:2164s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:11:28 (UTC+8)
2789tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:376s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:1983s | GPU cuda1:10477/24564MiB:99%;cuda2:4107/24564MiB:100%260820-14:08:27 (UTC+8)
2788tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:195s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:1802s | GPU cuda1:10477/24564MiB:100%;cuda2:4109/24564MiB:100%260820-14:05:26 (UTC+8)
2787tasks done=3792/12141 running=2 failed=0 skipped=0 pending=8347 | results done=2481/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s999:14s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:1621s | GPU cuda1:401/24564MiB:0%;cuda2:4109/24564MiB:100%260820-14:02:24 (UTC+8)
2786tasks done=3782/12141 running=2 failed=0 skipped=0 pending=8357 | results done=2472/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s888__gcn3:28s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:1439s | GPU cuda1:1637/24564MiB:84%;cuda2:4109/24564MiB:100%260820-13:59:23 (UTC+8)
2785tasks done=3778/12141 running=2 failed=0 skipped=0 pending=8361 | results done=2468/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s888__gat3:20s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:1258s | GPU cuda1:13091/24564MiB:93%;cuda2:4107/24564MiB:100%260820-13:56:22 (UTC+8)
2784tasks done=3775/12141 running=2 failed=0 skipped=0 pending=8364 | results done=2465/8095 failed=0 skipped=0 | RUNNING cuda1:external_downstream:run__LinguGKD__llama3_8b__wikics__s888__gat3:77s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:1077s | GPU cuda1:12035/24564MiB:54%;cuda2:4109/24564MiB:100%260820-13:53:21 (UTC+8)
2783tasks done=3774/12141 running=2 failed=0 skipped=0 pending=8365 | results done=2464/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s888__gat3:17s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:896s | GPU cuda1:13107/24564MiB:97%;cuda2:4109/24564MiB:100%260820-13:50:20 (UTC+8)
2782tasks done=3772/12141 running=2 failed=0 skipped=0 pending=8367 | results done=2462/8095 failed=0 skipped=0 | RUNNING cuda1:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s888__gat3:69s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:710s | GPU cuda1:13107/24564MiB:95%;cuda2:4109/24564MiB:100%260820-13:47:19 (UTC+8)
2781tasks done=3771/12141 running=2 failed=0 skipped=0 pending=8368 | results done=2461/8095 failed=0 skipped=0 | RUNNING cuda1:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s888__gat3:10s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:528s | GPU cuda1:13107/24564MiB:97%;cuda2:4109/24564MiB:100%260820-13:44:13 (UTC+8)
2780tasks done=3767/12141 running=2 failed=0 skipped=0 pending=8372 | results done=2457/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s888__gat2:49s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:347s | GPU cuda1:11203/24564MiB:95%;cuda2:4109/24564MiB:100%260820-13:41:11 (UTC+8)
2779tasks done=3764/12141 running=2 failed=0 skipped=0 pending=8375 | results done=2454/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s888__gat2:23s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s888:166s | GPU cuda1:9561/24564MiB:97%;cuda2:4109/24564MiB:100%260820-13:38:10 (UTC+8)
2778tasks done=3760/12141 running=2 failed=0 skipped=0 pending=8379 | results done=2452/8095 failed=0 skipped=0 | RUNNING cuda1:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s888__gat2:10s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:1772s | GPU cuda1:9561/24564MiB:93%;cuda2:16693/24564MiB:100%260820-13:35:09 (UTC+8)
2777tasks done=3756/12141 running=2 failed=0 skipped=0 pending=8383 | results done=2448/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s888__gat1:12s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:1591s | GPU cuda1:7693/24564MiB:93%;cuda2:16693/24564MiB:100%260820-13:32:08 (UTC+8)
2776tasks done=3750/12141 running=2 failed=0 skipped=0 pending=8389 | results done=2442/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s888__sage3:2s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:1409s | GPU cuda1:401/24564MiB:0%;cuda2:16693/24564MiB:100%260820-13:29:07 (UTC+8)
2775tasks done=3740/12141 running=2 failed=0 skipped=0 pending=8399 | results done=2432/8095 failed=0 skipped=0 | RUNNING cuda1:external_downstream:run__DuoKD__llama3_8b__wikics__s888__sage1:7s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:1228s | GPU cuda1:1053/24564MiB:64%;cuda2:16693/24564MiB:100%260820-13:26:05 (UTC+8)
2774tasks done=3730/12141 running=2 failed=0 skipped=0 pending=8409 | results done=2422/8095 failed=0 skipped=0 | RUNNING cuda1:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s888__gcn3:27s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:1047s | GPU cuda1:1637/24564MiB:80%;cuda2:16693/24564MiB:100%260820-13:23:04 (UTC+8)
2773tasks done=3720/12141 running=2 failed=0 skipped=0 pending=8419 | results done=2412/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s888__gcn2:20s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:866s | GPU cuda1:1553/24564MiB:79%;cuda2:16693/24564MiB:100%260820-13:20:03 (UTC+8)
2772tasks done=3707/12141 running=2 failed=0 skipped=0 pending=8432 | results done=2409/8095 failed=0 skipped=0 | RUNNING cuda1:student_oof:student_oof__wikics__s888__gat3:77s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:684s | GPU cuda1:12901/24564MiB:98%;cuda2:16691/24564MiB:100%260820-13:17:02 (UTC+8)
2771tasks done=3705/12141 running=2 failed=0 skipped=0 pending=8434 | results done=2409/8095 failed=0 skipped=0 | RUNNING cuda1:student_oof:student_oof__wikics__s888__gat1:44s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:503s | GPU cuda1:7669/24564MiB:97%;cuda2:16691/24564MiB:100%260820-13:14:00 (UTC+8)
2770tasks done=3698/12141 running=2 failed=0 skipped=0 pending=8441 | results done=2408/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s666__gat3:56s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:322s | GPU cuda1:13109/24564MiB:96%;cuda2:16691/24564MiB:100%260820-13:10:59 (UTC+8)
2769tasks done=3696/12141 running=2 failed=0 skipped=0 pending=8443 | results done=2406/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s666__gat3:120s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat3:141s | GPU cuda1:13107/24564MiB:96%;cuda2:16691/24564MiB:100%260820-13:07:58 (UTC+8)
2768tasks done=3694/12141 running=2 failed=0 skipped=0 pending=8445 | results done=2404/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s666__gat3:61s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s666__gat1:13s | GPU cuda1:13109/24564MiB:94%;cuda2:11277/24564MiB:100%260820-13:04:57 (UTC+8)
2767tasks done=3690/12141 running=2 failed=0 skipped=0 pending=8449 | results done=2400/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s666__gat3:61s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s666__gat1:23s | GPU cuda1:13107/24564MiB:96%;cuda2:11277/24564MiB:100%260820-13:01:56 (UTC+8)
2766tasks done=3686/12141 running=2 failed=0 skipped=0 pending=8453 | results done=2396/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s666__gat3:16s;cuda2:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s666__gat1:125s | GPU cuda1:13107/24564MiB:96%;cuda2:11277/24564MiB:100%260820-12:58:55 (UTC+8)
2765tasks done=3683/12141 running=2 failed=0 skipped=0 pending=8456 | results done=2393/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s666__gat3:80s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s666__gat1:55s | GPU cuda1:13109/24564MiB:96%;cuda2:11277/24564MiB:100%260820-12:55:54 (UTC+8)
2764tasks done=3678/12141 running=2 failed=0 skipped=0 pending=8461 | results done=2388/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s666__gat2:34s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s666__gat1:129s | GPU cuda1:9561/24564MiB:95%;cuda2:11277/24564MiB:100%260820-12:52:53 (UTC+8)
2763tasks done=3674/12141 running=2 failed=0 skipped=0 pending=8465 | results done=2384/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s666__gat2:31s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s666__gat1:90s | GPU cuda1:9561/24564MiB:95%;cuda2:11277/24564MiB:100%260820-12:49:52 (UTC+8)
2762tasks done=3670/12141 running=2 failed=0 skipped=0 pending=8469 | results done=2380/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s666__gat2:15s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s666__gat1:50s | GPU cuda1:9561/24564MiB:95%;cuda2:11277/24564MiB:100%260820-12:46:51 (UTC+8)
2761tasks done=3657/12141 running=2 failed=0 skipped=0 pending=8482 | results done=2367/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s666__sage3:5s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s666__sage1:22s | GPU cuda1:1439/24564MiB:73%;cuda2:4635/24564MiB:100%260820-12:43:50 (UTC+8)
2760tasks done=3642/12141 running=2 failed=0 skipped=0 pending=8497 | results done=2352/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s666__sage2:11s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s666__sage1:16s | GPU cuda1:1435/24564MiB:72%;cuda2:4635/24564MiB:100%260820-12:40:49 (UTC+8)
2759tasks done=3633/12141 running=2 failed=0 skipped=0 pending=8506 | results done=2344/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s666__gcn3:3s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat2:1156s | GPU cuda1:577/24564MiB:3%;cuda2:13145/24564MiB:100%260820-12:37:47 (UTC+8)
2758tasks done=3624/12141 running=2 failed=0 skipped=0 pending=8515 | results done=2335/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s666__gcn1:9s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat2:975s | GPU cuda1:1509/24564MiB:0%;cuda2:13145/24564MiB:100%260820-12:34:46 (UTC+8)
2757tasks done=3611/12141 running=2 failed=0 skipped=0 pending=8528 | results done=2322/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__J_Labels50__llama3_8b__wikics__s666__gcn2:10s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat2:794s | GPU cuda1:1553/24564MiB:84%;cuda2:13145/24564MiB:100%260820-12:31:45 (UTC+8)
2756tasks done=3602/12141 running=2 failed=0 skipped=0 pending=8537 | results done=2313/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__C_RawUtility__llama3_8b__wikics__s666__gcn2:8s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat2:613s | GPU cuda1:1553/24564MiB:76%;cuda2:13145/24564MiB:100%260820-12:28:44 (UTC+8)
2755tasks done=3594/12141 running=2 failed=0 skipped=0 pending=8545 | results done=2305/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__GlobalOnly__llama3_8b__wikics__s666__gcn2:4s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat2:431s | GPU cuda1:1553/24564MiB:34%;cuda2:13145/24564MiB:100%260820-12:25:43 (UTC+8)
2754tasks done=3588/12141 running=2 failed=0 skipped=0 pending=8551 | results done=2299/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s666__gcn2:9s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat2:250s | GPU cuda1:1553/24564MiB:78%;cuda2:13145/24564MiB:100%260820-12:22:41 (UTC+8)
2753tasks done=3580/12141 running=2 failed=0 skipped=0 pending=8559 | results done=2291/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s666__gcn2:1s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat2:69s | GPU cuda1:14/24564MiB:0%;cuda2:13145/24564MiB:100%260820-12:19:40 (UTC+8)
2752tasks done=3577/12141 running=2 failed=0 skipped=0 pending=8562 | results done=2290/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:5018s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat1:620s | GPU cuda1:525/24564MiB:12%;cuda2:11277/24564MiB:100%260820-12:16:39 (UTC+8)
2751tasks done=3577/12141 running=2 failed=0 skipped=0 pending=8562 | results done=2290/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:4837s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat1:438s | GPU cuda1:525/24564MiB:16%;cuda2:11277/24564MiB:100%260820-12:13:38 (UTC+8)
2750tasks done=3577/12141 running=2 failed=0 skipped=0 pending=8562 | results done=2290/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:4656s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat1:257s | GPU cuda1:525/24564MiB:15%;cuda2:11277/24564MiB:100%260820-12:10:36 (UTC+8)
2749tasks done=3577/12141 running=2 failed=0 skipped=0 pending=8562 | results done=2290/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:4475s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gat1:76s | GPU cuda1:525/24564MiB:14%;cuda2:11277/24564MiB:100%260820-12:07:35 (UTC+8)
2748tasks done=3576/12141 running=2 failed=0 skipped=0 pending=8563 | results done=2290/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:4294s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__sage3:427s | GPU cuda1:525/24564MiB:15%;cuda2:5025/24564MiB:100%260820-12:04:34 (UTC+8)
2747tasks done=3576/12141 running=2 failed=0 skipped=0 pending=8563 | results done=2290/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:4112s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__sage3:245s | GPU cuda1:525/24564MiB:15%;cuda2:5025/24564MiB:100%260820-12:01:33 (UTC+8)
2746tasks done=3576/12141 running=2 failed=0 skipped=0 pending=8563 | results done=2290/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:3931s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__sage3:64s | GPU cuda1:525/24564MiB:13%;cuda2:5025/24564MiB:100%260820-11:58:31 (UTC+8)
2745tasks done=3575/12141 running=2 failed=0 skipped=0 pending=8564 | results done=2290/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:3750s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__sage2:316s | GPU cuda1:525/24564MiB:12%;cuda2:5019/24564MiB:100%260820-11:55:30 (UTC+8)
2744tasks done=3575/12141 running=2 failed=0 skipped=0 pending=8564 | results done=2290/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:3569s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__sage2:134s | GPU cuda1:523/24564MiB:15%;cuda2:5019/24564MiB:100%260820-11:52:29 (UTC+8)
2743tasks done=3574/12141 running=2 failed=0 skipped=0 pending=8565 | results done=2289/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:3388s;cuda2:task_only:taskonly__wikics__s888__gat3:36s | GPU cuda1:523/24564MiB:16%;cuda2:16693/24564MiB:100%260820-11:49:28 (UTC+8)
2742tasks done=3573/12141 running=2 failed=0 skipped=0 pending=8566 | results done=2289/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:3207s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__sage1:177s | GPU cuda1:521/24564MiB:0%;cuda2:4635/24564MiB:100%260820-11:46:27 (UTC+8)
2741tasks done=3572/12141 running=2 failed=0 skipped=0 pending=8567 | results done=2288/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:3026s;cuda2:task_only:taskonly__wikics__s888__gat2:61s | GPU cuda1:525/24564MiB:16%;cuda2:13143/24564MiB:100%260820-11:43:26 (UTC+8)
2740tasks done=3571/12141 running=2 failed=0 skipped=0 pending=8568 | results done=2288/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:2845s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gcn3:541s | GPU cuda1:525/24564MiB:14%;cuda2:5221/24564MiB:100%260820-11:40:25 (UTC+8)
2739tasks done=3571/12141 running=2 failed=0 skipped=0 pending=8568 | results done=2288/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:2664s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gcn3:360s | GPU cuda1:525/24564MiB:13%;cuda2:5221/24564MiB:100%260820-11:37:24 (UTC+8)
2738tasks done=3571/12141 running=2 failed=0 skipped=0 pending=8568 | results done=2288/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:2483s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gcn3:179s | GPU cuda1:525/24564MiB:14%;cuda2:5221/24564MiB:100%260820-11:34:23 (UTC+8)
2737tasks done=3570/12141 running=2 failed=0 skipped=0 pending=8569 | results done=2287/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:2302s;cuda2:task_only:taskonly__wikics__s888__gat1:36s | GPU cuda1:525/24564MiB:15%;cuda2:11277/24564MiB:100%260820-11:31:22 (UTC+8)
2736tasks done=3569/12141 running=2 failed=0 skipped=0 pending=8570 | results done=2287/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:2120s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gcn2:370s | GPU cuda1:525/24564MiB:15%;cuda2:5137/24564MiB:100%260820-11:28:21 (UTC+8)
2735tasks done=3569/12141 running=2 failed=0 skipped=0 pending=8570 | results done=2287/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:1939s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gcn2:188s | GPU cuda1:525/24564MiB:16%;cuda2:5137/24564MiB:100%260820-11:25:19 (UTC+8)
2734tasks done=3569/12141 running=2 failed=0 skipped=0 pending=8570 | results done=2287/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:1758s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gcn2:7s | GPU cuda1:525/24564MiB:10%;cuda2:5137/24564MiB:100%260820-11:22:18 (UTC+8)
2733tasks done=3567/12141 running=2 failed=0 skipped=0 pending=8572 | results done=2286/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:1577s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gcn1:237s | GPU cuda1:525/24564MiB:16%;cuda2:5093/24564MiB:100%260820-11:19:17 (UTC+8)
2732tasks done=3567/12141 running=2 failed=0 skipped=0 pending=8572 | results done=2286/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:1396s;cuda2:utility_predictions:utility__llama3_8b__wikics__s888__gcn1:56s | GPU cuda1:525/24564MiB:16%;cuda2:5093/24564MiB:100%260820-11:16:16 (UTC+8)
2731tasks done=3562/12141 running=2 failed=0 skipped=0 pending=8577 | results done=2281/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:1214s;cuda2:task_only:taskonly__wikics__s888__gcn1:10s | GPU cuda1:525/24564MiB:13%;cuda2:5093/24564MiB:100%260820-11:13:15 (UTC+8)
2730tasks done=3561/12141 running=2 failed=0 skipped=0 pending=8578 | results done=2281/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:1033s;cuda2:trace_analysis:trace_analysis__llama3_8b__wikics__s888:2s | GPU cuda1:525/24564MiB:17%;cuda2:3985/24564MiB:100%260820-11:10:13 (UTC+8)
2729tasks done=3560/12141 running=2 failed=0 skipped=0 pending=8579 | results done=2280/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:852s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s42__gat2:64s | GPU cuda1:525/24564MiB:17%;cuda2:13145/24564MiB:100%260820-11:07:12 (UTC+8)
2728tasks done=3559/12141 running=2 failed=0 skipped=0 pending=8580 | results done=2279/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:671s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s42__gat2:126s | GPU cuda1:525/24564MiB:15%;cuda2:13145/24564MiB:100%260820-11:04:11 (UTC+8)
2727tasks done=3558/12141 running=2 failed=0 skipped=0 pending=8581 | results done=2278/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:489s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s42__gat2:188s | GPU cuda1:525/24564MiB:14%;cuda2:13145/24564MiB:100%260820-11:01:10 (UTC+8)
2726tasks done=3558/12141 running=2 failed=0 skipped=0 pending=8581 | results done=2278/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:308s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s42__gat2:7s | GPU cuda1:525/24564MiB:18%;cuda2:13143/24564MiB:100%260820-10:58:09 (UTC+8)
2725tasks done=3557/12141 running=2 failed=0 skipped=0 pending=8582 | results done=2277/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_bundle:hcr_bundle__llama3_8b__wikics__s666:126s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s42__gat2:68s | GPU cuda1:525/24564MiB:18%;cuda2:13145/24564MiB:100%260820-10:55:08 (UTC+8)
2724tasks done=3545/12141 running=2 failed=0 skipped=0 pending=8594 | results done=2275/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s42__gat3:77s;cuda2:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s42__gat2:127s | GPU cuda1:13109/24564MiB:96%;cuda2:13145/24564MiB:100%260820-10:52:06 (UTC+8)
2723tasks done=3543/12141 running=2 failed=0 skipped=0 pending=8596 | results done=2273/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s42__gat3:18s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s42__gat2:190s | GPU cuda1:13107/24564MiB:96%;cuda2:13145/24564MiB:100%260820-10:49:04 (UTC+8)
2722tasks done=3541/12141 running=2 failed=0 skipped=0 pending=8598 | results done=2271/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s42__gat3:82s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s42__gat2:9s | GPU cuda1:13109/24564MiB:96%;cuda2:13145/24564MiB:100%260820-10:46:03 (UTC+8)
2721tasks done=3538/12141 running=2 failed=0 skipped=0 pending=8601 | results done=2268/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s42__gat3:82s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s42__gat2:72s | GPU cuda1:13109/24564MiB:94%;cuda2:13145/24564MiB:100%260820-10:43:02 (UTC+8)
2720tasks done=3536/12141 running=2 failed=0 skipped=0 pending=8603 | results done=2266/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s42__gat3:23s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s42__gat2:46s | GPU cuda1:13109/24564MiB:94%;cuda2:13145/24564MiB:100%260820-10:40:01 (UTC+8)
2719tasks done=3532/12141 running=2 failed=0 skipped=0 pending=8607 | results done=2262/8095 failed=0 skipped=0 | RUNNING cuda1:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s42__gat3:53s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s42__gat2:238s | GPU cuda1:13109/24564MiB:96%;cuda2:13147/24564MiB:100%260820-10:37:00 (UTC+8)
2718tasks done=3530/12141 running=2 failed=0 skipped=0 pending=8609 | results done=2260/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s42__gat3:57s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s42__gat2:57s | GPU cuda1:13091/24564MiB:96%;cuda2:13147/24564MiB:100%260820-10:33:59 (UTC+8)
2717tasks done=3528/12141 running=2 failed=0 skipped=0 pending=8611 | results done=2258/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s42__gat3:0s;cuda2:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s42__gat2:127s | GPU cuda1:14/24564MiB:0%;cuda2:14787/24564MiB:100%260820-10:30:57 (UTC+8)
2716tasks done=3524/12141 running=2 failed=0 skipped=0 pending=8615 | results done=2254/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s42__gat3:38s;cuda2:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s42__gat2:197s | GPU cuda1:13109/24564MiB:94%;cuda2:14787/24564MiB:100%260820-10:27:56 (UTC+8)
2715tasks done=3522/12141 running=2 failed=0 skipped=0 pending=8617 | results done=2252/8095 failed=0 skipped=0 | RUNNING cuda1:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s42__gat3:102s;cuda2:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s42__gat2:16s | GPU cuda1:13109/24564MiB:94%;cuda2:14787/24564MiB:100%260820-10:24:55 (UTC+8)
2714tasks done=3519/12141 running=2 failed=0 skipped=0 pending=8620 | results done=2249/8095 failed=0 skipped=0 | RUNNING cuda1:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s42__gat3:43s;cuda2:external_downstream:run__LinguGKD__llama3_8b__wikics__s42__gat2:35s | GPU cuda1:13107/24564MiB:97%;cuda2:15313/24564MiB:100%260820-10:21:54 (UTC+8)
2713tasks done=3516/12141 running=2 failed=0 skipped=0 pending=8623 | results done=2248/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:41591s;cuda2:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s42__gat2:97s | GPU cuda1:7817/24564MiB:53%;cuda2:13145/24564MiB:100%260820-10:18:52 (UTC+8)
2712tasks done=3515/12141 running=2 failed=0 skipped=0 pending=8624 | results done=2247/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:41410s;cuda2:trace_downstream:run__T3_TeacherOptGlobal__llama3_8b__wikics__s42__gat2:100s | GPU cuda1:7817/24564MiB:53%;cuda2:13145/24564MiB:100%260820-10:15:51 (UTC+8)
2711tasks done=3514/12141 running=2 failed=0 skipped=0 pending=8625 | results done=2246/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:41229s;cuda2:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s42__gat2:162s | GPU cuda1:7817/24564MiB:54%;cuda2:13145/24564MiB:100%260820-10:12:50 (UTC+8)
2710tasks done=3513/12141 running=2 failed=0 skipped=0 pending=8626 | results done=2245/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:41046s;cuda2:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s42__gat2:223s | GPU cuda1:7817/24564MiB:53%;cuda2:13145/24564MiB:100%260820-10:09:49 (UTC+8)
2709tasks done=3513/12141 running=2 failed=0 skipped=0 pending=8626 | results done=2245/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:40865s;cuda2:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s42__gat2:42s | GPU cuda1:7817/24564MiB:54%;cuda2:13145/24564MiB:100%260820-10:06:46 (UTC+8)
2708tasks done=3512/12141 running=2 failed=0 skipped=0 pending=8627 | results done=2244/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:40684s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s42__gat1:2s | GPU cuda1:7817/24564MiB:54%;cuda2:3985/24564MiB:100%260820-10:03:45 (UTC+8)
2707tasks done=3509/12141 running=2 failed=0 skipped=0 pending=8630 | results done=2241/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:40503s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s42__gat1:45s | GPU cuda1:7817/24564MiB:52%;cuda2:11277/24564MiB:100%260820-10:00:44 (UTC+8)
2706tasks done=3507/12141 running=2 failed=0 skipped=0 pending=8632 | results done=2239/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:40322s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s42__gat1:2s | GPU cuda1:7817/24564MiB:51%;cuda2:3985/24564MiB:100%260820-09:57:43 (UTC+8)
2705tasks done=3505/12141 running=2 failed=0 skipped=0 pending=8634 | results done=2237/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:40141s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s42__gat1:32s | GPU cuda1:7817/24564MiB:54%;cuda2:11277/24564MiB:100%260820-09:54:42 (UTC+8)
2704tasks done=3502/12141 running=2 failed=0 skipped=0 pending=8637 | results done=2234/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:39960s;cuda2:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s42__gat1:58s | GPU cuda1:7817/24564MiB:51%;cuda2:11279/24564MiB:100%260820-09:51:41 (UTC+8)
2703tasks done=3501/12141 running=2 failed=0 skipped=0 pending=8638 | results done=2233/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:39779s;cuda2:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s42__gat1:26s | GPU cuda1:7817/24564MiB:53%;cuda2:11279/24564MiB:100%260820-09:48:40 (UTC+8)
2702tasks done=3498/12141 running=2 failed=0 skipped=0 pending=8641 | results done=2230/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:39598s;cuda2:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s42__gat1:27s | GPU cuda1:7817/24564MiB:51%;cuda2:11277/24564MiB:100%260820-09:45:39 (UTC+8)
2701tasks done=3495/12141 running=2 failed=0 skipped=0 pending=8644 | results done=2227/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:39417s;cuda2:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s42__gat1:54s | GPU cuda1:7817/24564MiB:51%;cuda2:11277/24564MiB:100%260820-09:42:38 (UTC+8)
2700tasks done=3493/12141 running=2 failed=0 skipped=0 pending=8646 | results done=2225/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:39236s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s42__sage3:36s | GPU cuda1:7817/24564MiB:51%;cuda2:5025/24564MiB:100%260820-09:39:37 (UTC+8)
2699tasks done=3491/12141 running=2 failed=0 skipped=0 pending=8648 | results done=2223/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:39055s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s42__sage3:23s | GPU cuda1:7817/24564MiB:54%;cuda2:5025/24564MiB:100%260820-09:36:36 (UTC+8)
2698tasks done=3488/12141 running=2 failed=0 skipped=0 pending=8651 | results done=2220/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:38874s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s42__sage3:68s | GPU cuda1:7817/24564MiB:54%;cuda2:5025/24564MiB:100%260820-09:33:35 (UTC+8)
2697tasks done=3486/12141 running=2 failed=0 skipped=0 pending=8653 | results done=2218/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:38693s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s42__sage3:26s | GPU cuda1:7817/24564MiB:49%;cuda2:5025/24564MiB:100%260820-09:30:34 (UTC+8)
2696tasks done=3483/12141 running=2 failed=0 skipped=0 pending=8656 | results done=2215/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:38512s;cuda2:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s42__sage3:97s | GPU cuda1:7817/24564MiB:53%;cuda2:4785/24564MiB:100%260820-09:27:33 (UTC+8)
2695tasks done=3481/12141 running=2 failed=0 skipped=0 pending=8658 | results done=2213/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:38331s;cuda2:external_downstream:run__LinguGKD__llama3_8b__wikics__s42__sage3:7s | GPU cuda1:7817/24564MiB:53%;cuda2:5493/24564MiB:100%260820-09:24:32 (UTC+8)
2694tasks done=3478/12141 running=2 failed=0 skipped=0 pending=8661 | results done=2210/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:38149s;cuda2:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s42__sage3:63s | GPU cuda1:7817/24564MiB:54%;cuda2:5025/24564MiB:100%260820-09:21:31 (UTC+8)
2693tasks done=3476/12141 running=2 failed=0 skipped=0 pending=8663 | results done=2208/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:37968s;cuda2:hcr_downstream:run__R3_BudgetMatched__llama3_8b__wikics__s42__sage2:35s | GPU cuda1:7817/24564MiB:53%;cuda2:5019/24564MiB:100%260820-09:18:29 (UTC+8)
2692tasks done=3474/12141 running=2 failed=0 skipped=0 pending=8665 | results done=2206/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:37787s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s42__sage2:12s | GPU cuda1:7817/24564MiB:50%;cuda2:5019/24564MiB:100%260820-09:15:28 (UTC+8)
2691tasks done=3471/12141 running=2 failed=0 skipped=0 pending=8668 | results done=2203/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:37606s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s42__sage2:50s | GPU cuda1:7817/24564MiB:54%;cuda2:5019/24564MiB:100%260820-09:12:28 (UTC+8)
2690tasks done=3469/12141 running=2 failed=0 skipped=0 pending=8670 | results done=2201/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:37425s;cuda2:hcr_downstream:run__HCR_RegionOnly__llama3_8b__wikics__s42__sage2:18s | GPU cuda1:7817/24564MiB:53%;cuda2:5019/24564MiB:100%260820-09:09:26 (UTC+8)
2689tasks done=3466/12141 running=2 failed=0 skipped=0 pending=8673 | results done=2198/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:37244s;cuda2:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s42__sage2:78s | GPU cuda1:7817/24564MiB:53%;cuda2:4779/24564MiB:100%260820-09:06:25 (UTC+8)
2688tasks done=3464/12141 running=2 failed=0 skipped=0 pending=8675 | results done=2196/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:37063s;cuda2:external_downstream:run__DuoKD__llama3_8b__wikics__s42__sage2:16s | GPU cuda1:7817/24564MiB:53%;cuda2:5021/24564MiB:100%260820-09:03:24 (UTC+8)
2687tasks done=3461/12141 running=2 failed=0 skipped=0 pending=8678 | results done=2193/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:36882s;cuda2:trace_downstream:run__T3_TeacherOptGlobal__llama3_8b__wikics__s42__sage2:46s | GPU cuda1:7817/24564MiB:54%;cuda2:5019/24564MiB:100%260820-09:00:23 (UTC+8)
2686tasks done=3459/12141 running=2 failed=0 skipped=0 pending=8680 | results done=2191/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:36700s;cuda2:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s42__sage2:34s | GPU cuda1:7817/24564MiB:54%;cuda2:5019/24564MiB:100%260820-08:57:22 (UTC+8)
2685tasks done=3456/12141 running=2 failed=0 skipped=0 pending=8683 | results done=2188/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:36519s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s42__sage1:10s | GPU cuda1:7817/24564MiB:54%;cuda2:4635/24564MiB:100%260820-08:54:20 (UTC+8)
2684tasks done=3453/12141 running=2 failed=0 skipped=0 pending=8686 | results done=2185/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:36338s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s42__sage1:8s | GPU cuda1:7817/24564MiB:53%;cuda2:4635/24564MiB:100%260820-08:51:19 (UTC+8)
2683tasks done=3449/12141 running=2 failed=0 skipped=0 pending=8690 | results done=2181/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:36157s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s42__sage1:59s | GPU cuda1:7817/24564MiB:53%;cuda2:4635/24564MiB:100%260820-08:48:18 (UTC+8)
2682tasks done=3447/12141 running=2 failed=0 skipped=0 pending=8692 | results done=2179/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:35976s;cuda2:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s42__sage1:3s | GPU cuda1:7817/24564MiB:53%;cuda2:4283/24564MiB:100%260820-08:45:17 (UTC+8)
2681tasks done=3442/12141 running=2 failed=0 skipped=0 pending=8697 | results done=2174/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:35794s;cuda2:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s42__sage1:2s | GPU cuda1:7817/24564MiB:54%;cuda2:3985/24564MiB:100%260820-08:42:16 (UTC+8)
2680tasks done=3439/12141 running=2 failed=0 skipped=0 pending=8700 | results done=2171/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:35613s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s42__gcn3:129s | GPU cuda1:7817/24564MiB:53%;cuda2:5221/24564MiB:100%260820-08:39:15 (UTC+8)
2679tasks done=3438/12141 running=2 failed=0 skipped=0 pending=8701 | results done=2170/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:35432s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s42__gcn3:78s | GPU cuda1:7817/24564MiB:53%;cuda2:5221/24564MiB:100%260820-08:36:14 (UTC+8)
2678tasks done=3437/12141 running=2 failed=0 skipped=0 pending=8702 | results done=2169/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:35251s;cuda2:hcr_downstream:run__R0_RandomRoute__llama3_8b__wikics__s42__gcn3:27s | GPU cuda1:7817/24564MiB:82%;cuda2:5221/24564MiB:100%260820-08:33:13 (UTC+8)
2677tasks done=3435/12141 running=2 failed=0 skipped=0 pending=8704 | results done=2167/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:35070s;cuda2:hcr_downstream:run__HCR_Transfer__llama3_8b__wikics__s42__gcn3:106s | GPU cuda1:7817/24564MiB:54%;cuda2:5219/24564MiB:100%260820-08:30:12 (UTC+8)
2676tasks done=3434/12141 running=2 failed=0 skipped=0 pending=8705 | results done=2166/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:34889s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s42__gcn3:55s | GPU cuda1:7817/24564MiB:54%;cuda2:5221/24564MiB:100%260820-08:27:10 (UTC+8)
2675tasks done=3432/12141 running=2 failed=0 skipped=0 pending=8707 | results done=2164/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:34707s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s42__gcn3:129s | GPU cuda1:7817/24564MiB:53%;cuda2:5221/24564MiB:100%260820-08:24:09 (UTC+8)
2674tasks done=3431/12141 running=2 failed=0 skipped=0 pending=8708 | results done=2163/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:34526s;cuda2:hcr_downstream:run__HCR_GlobalOnly__llama3_8b__wikics__s42__gcn3:78s | GPU cuda1:7817/24564MiB:54%;cuda2:5221/24564MiB:100%260820-08:21:08 (UTC+8)
2673tasks done=3430/12141 running=2 failed=0 skipped=0 pending=8709 | results done=2162/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:34345s;cuda2:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s42__gcn3:29s | GPU cuda1:7817/24564MiB:53%;cuda2:5219/24564MiB:100%260820-08:18:07 (UTC+8)
2672tasks done=3428/12141 running=2 failed=0 skipped=0 pending=8711 | results done=2160/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:34164s;cuda2:external_downstream:run__DuoKD__llama3_8b__wikics__s42__gcn3:39s | GPU cuda1:7817/24564MiB:53%;cuda2:5223/24564MiB:100%260820-08:15:06 (UTC+8)
2671tasks done=3426/12141 running=2 failed=0 skipped=0 pending=8713 | results done=2158/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:33983s;cuda2:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s42__gcn3:64s | GPU cuda1:7817/24564MiB:53%;cuda2:5221/24564MiB:100%260820-08:12:05 (UTC+8)
2670tasks done=3425/12141 running=2 failed=0 skipped=0 pending=8714 | results done=2157/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:33802s;cuda2:trace_downstream:run__T3_TeacherOptGlobal__llama3_8b__wikics__s42__gcn3:13s | GPU cuda1:7817/24564MiB:51%;cuda2:5219/24564MiB:100%260820-08:09:03 (UTC+8)
2669tasks done=3423/12141 running=2 failed=0 skipped=0 pending=8716 | results done=2155/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:33621s;cuda2:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s42__gcn3:92s | GPU cuda1:7817/24564MiB:53%;cuda2:5221/24564MiB:100%260820-08:06:02 (UTC+8)
2668tasks done=3421/12141 running=2 failed=0 skipped=0 pending=8718 | results done=2153/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:33440s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s42__gcn1:17s | GPU cuda1:7817/24564MiB:54%;cuda2:5093/24564MiB:100%260820-08:03:01 (UTC+8)
2667tasks done=3418/12141 running=2 failed=0 skipped=0 pending=8721 | results done=2150/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:33259s;cuda2:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s42__gcn1:25s | GPU cuda1:7817/24564MiB:53%;cuda2:5093/24564MiB:100%260820-08:00:00 (UTC+8)
2666tasks done=3414/12141 running=2 failed=0 skipped=0 pending=8725 | results done=2146/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:33077s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s42__gcn1:6s | GPU cuda1:7817/24564MiB:51%;cuda2:5093/24564MiB:100%260820-07:56:59 (UTC+8)
2665tasks done=3409/12141 running=2 failed=0 skipped=0 pending=8730 | results done=2141/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:32896s;cuda2:external_downstream:run__LinguGKD__llama3_8b__wikics__s42__gcn1:23s | GPU cuda1:7817/24564MiB:54%;cuda2:5483/24564MiB:100%260820-07:53:57 (UTC+8)
2664tasks done=3406/12141 running=2 failed=0 skipped=0 pending=8733 | results done=2138/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:32715s;cuda2:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s42__gcn1:9s | GPU cuda1:7817/24564MiB:53%;cuda2:5093/24564MiB:100%260820-07:50:56 (UTC+8)
2663tasks done=3403/12141 running=2 failed=0 skipped=0 pending=8736 | results done=2135/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:32534s;cuda2:hcr_downstream:run__E4_ReliabilityFeature__llama3_8b__wikics__s42__gcn2:75s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-07:47:55 (UTC+8)
2662tasks done=3401/12141 running=2 failed=0 skipped=0 pending=8738 | results done=2133/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:32353s;cuda2:hcr_downstream:run__E2_NoContextMetadata__llama3_8b__wikics__s42__gcn2:96s | GPU cuda1:7817/24564MiB:51%;cuda2:5137/24564MiB:100%260820-07:44:54 (UTC+8)
2661tasks done=3400/12141 running=2 failed=0 skipped=0 pending=8739 | results done=2132/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:32171s;cuda2:hcr_downstream:run__E1_Snapshot3__llama3_8b__wikics__s42__gcn2:16s | GPU cuda1:7817/24564MiB:53%;cuda2:5137/24564MiB:100%260820-07:41:53 (UTC+8)
2660tasks done=3398/12141 running=2 failed=0 skipped=0 pending=8741 | results done=2130/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:31990s;cuda2:hcr_downstream:run__J_Labels50__llama3_8b__wikics__s42__gcn2:38s | GPU cuda1:7817/24564MiB:53%;cuda2:5137/24564MiB:100%260820-07:38:51 (UTC+8)
2659tasks done=3396/12141 running=2 failed=0 skipped=0 pending=8743 | results done=2128/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:31809s;cuda2:hcr_downstream:run__J_Labels10__llama3_8b__wikics__s42__gcn2:58s | GPU cuda1:7817/24564MiB:53%;cuda2:5137/24564MiB:100%260820-07:35:50 (UTC+8)
2658tasks done=3394/12141 running=2 failed=0 skipped=0 pending=8745 | results done=2126/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:31628s;cuda2:hcr_downstream:run__G_ProbeT1__llama3_8b__wikics__s42__gcn2:78s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-07:32:49 (UTC+8)
2657tasks done=3392/12141 running=2 failed=0 skipped=0 pending=8747 | results done=2124/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:31447s;cuda2:hcr_downstream:run__F_EtaQuarter__llama3_8b__wikics__s42__gcn2:100s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-07:29:48 (UTC+8)
2656tasks done=3391/12141 running=2 failed=0 skipped=0 pending=8748 | results done=2123/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:31265s;cuda2:hcr_downstream:run__E_NoArchitecture__llama3_8b__wikics__s42__gcn2:20s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-07:26:47 (UTC+8)
2655tasks done=3389/12141 running=2 failed=0 skipped=0 pending=8750 | results done=2121/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:31084s;cuda2:hcr_downstream:run__C_RawUtility__llama3_8b__wikics__s42__gcn2:40s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-07:23:46 (UTC+8)
2654tasks done=3387/12141 running=2 failed=0 skipped=0 pending=8752 | results done=2119/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:30903s;cuda2:hcr_downstream:run__A_RemoveGlobal__llama3_8b__wikics__s42__gcn2:61s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-07:20:45 (UTC+8)
2653tasks done=3385/12141 running=2 failed=0 skipped=0 pending=8754 | results done=2117/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:30722s;cuda2:hcr_downstream:run__HCR_EB_NoRegion__llama3_8b__wikics__s42__gcn2:80s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-07:17:43 (UTC+8)
2652tasks done=3384/12141 running=2 failed=0 skipped=0 pending=8755 | results done=2116/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:30541s;cuda2:hcr_downstream:run__HCR_EB_NoArch__llama3_8b__wikics__s42__gcn2:0s | GPU cuda1:7817/24564MiB:51%;cuda2:3597/24564MiB:100%260820-07:14:42 (UTC+8)
2651tasks done=3382/12141 running=2 failed=0 skipped=0 pending=8757 | results done=2114/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:30359s;cuda2:hcr_downstream:run__LocalOnly_Transfer__llama3_8b__wikics__s42__gcn2:20s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-07:11:41 (UTC+8)
2650tasks done=3380/12141 running=2 failed=0 skipped=0 pending=8759 | results done=2112/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:30178s;cuda2:hcr_downstream:run__TRIP_HCR__llama3_8b__wikics__s42__gcn2:25s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-07:08:39 (UTC+8)
2649tasks done=3378/12141 running=2 failed=0 skipped=0 pending=8761 | results done=2110/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:29997s;cuda2:hcr_downstream:run__HCR_Teacher__llama3_8b__wikics__s42__gcn2:46s | GPU cuda1:7817/24564MiB:53%;cuda2:5137/24564MiB:100%260820-07:05:38 (UTC+8)
2648tasks done=3377/12141 running=2 failed=0 skipped=0 pending=8762 | results done=2109/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:29816s;cuda2:rci_downstream:run__RCI_Unconditioned__llama3_8b__wikics__s42__gcn2:0s | GPU cuda1:7817/24564MiB:54%;cuda2:3597/24564MiB:100%260820-07:02:37 (UTC+8)
2647tasks done=3376/12141 running=2 failed=0 skipped=0 pending=8763 | results done=2108/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:29635s;cuda2:rci_downstream:run__RCI_Static__llama3_8b__wikics__s42__gcn2:80s | GPU cuda1:7817/24564MiB:54%;cuda2:5133/24564MiB:100%260820-06:59:36 (UTC+8)
2646tasks done=3374/12141 running=2 failed=0 skipped=0 pending=8765 | results done=2106/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:29454s;cuda2:hcr_downstream:run__R2_ShuffledResidual__llama3_8b__wikics__s42__gcn2:95s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-06:56:35 (UTC+8)
2645tasks done=3373/12141 running=2 failed=0 skipped=0 pending=8766 | results done=2105/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:29273s;cuda2:hcr_downstream:run__R1_HistogramRoute__llama3_8b__wikics__s42__gcn2:15s | GPU cuda1:7817/24564MiB:89%;cuda2:5137/24564MiB:100%260820-06:53:34 (UTC+8)
2644tasks done=3371/12141 running=2 failed=0 skipped=0 pending=8768 | results done=2103/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:29092s;cuda2:hcr_downstream:run__HCR_EB__llama3_8b__wikics__s42__gcn2:35s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-06:50:33 (UTC+8)
2643tasks done=3369/12141 running=2 failed=0 skipped=0 pending=8770 | results done=2101/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:28911s;cuda2:hcr_downstream:run__HCR_NodeOnly__llama3_8b__wikics__s42__gcn2:57s | GPU cuda1:7817/24564MiB:50%;cuda2:5137/24564MiB:100%260820-06:47:32 (UTC+8)
2642tasks done=3367/12141 running=2 failed=0 skipped=0 pending=8772 | results done=2099/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:28729s;cuda2:hcr_downstream:run__HCR_ArchOnly__llama3_8b__wikics__s42__gcn2:78s | GPU cuda1:7817/24564MiB:50%;cuda2:5137/24564MiB:100%260820-06:44:31 (UTC+8)
2641tasks done=3365/12141 running=2 failed=0 skipped=0 pending=8774 | results done=2097/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:28548s;cuda2:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s42__gcn2:86s | GPU cuda1:7817/24564MiB:54%;cuda2:5217/24564MiB:100%260820-06:41:29 (UTC+8)
2640tasks done=3364/12141 running=2 failed=0 skipped=0 pending=8775 | results done=2096/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:28367s;cuda2:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s42__gcn2:13s | GPU cuda1:7817/24564MiB:50%;cuda2:5217/24564MiB:100%260820-06:38:28 (UTC+8)
2639tasks done=3361/12141 running=2 failed=0 skipped=0 pending=8778 | results done=2093/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:28186s;cuda2:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s42__gcn2:39s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-06:35:27 (UTC+8)
2638tasks done=3358/12141 running=2 failed=0 skipped=0 pending=8781 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:28004s;cuda2:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s42__gcn2:97s | GPU cuda1:7817/24564MiB:54%;cuda2:5137/24564MiB:100%260820-06:32:26 (UTC+8)
2637tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:27823s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:18585s | GPU cuda1:7817/24564MiB:54%;cuda2:4107/24564MiB:100%260820-06:29:24 (UTC+8)
2636tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:27642s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:18404s | GPU cuda1:7817/24564MiB:52%;cuda2:4109/24564MiB:100%260820-06:26:23 (UTC+8)
2635tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:27461s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:18222s | GPU cuda1:7817/24564MiB:53%;cuda2:4109/24564MiB:100%260820-06:23:22 (UTC+8)
2634tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:27280s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:18041s | GPU cuda1:7817/24564MiB:52%;cuda2:4109/24564MiB:100%260820-06:20:21 (UTC+8)
2633tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:27098s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:17860s | GPU cuda1:7817/24564MiB:51%;cuda2:4107/24564MiB:100%260820-06:17:20 (UTC+8)
2632tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:26917s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:17679s | GPU cuda1:7817/24564MiB:50%;cuda2:4109/24564MiB:100%260820-06:14:18 (UTC+8)
2631tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:26736s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:17498s | GPU cuda1:7817/24564MiB:54%;cuda2:4109/24564MiB:100%260820-06:11:17 (UTC+8)
2630tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:26555s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:17317s | GPU cuda1:7817/24564MiB:53%;cuda2:4109/24564MiB:100%260820-06:08:16 (UTC+8)
2629tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:26373s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:17135s | GPU cuda1:7817/24564MiB:50%;cuda2:4109/24564MiB:100%260820-06:05:15 (UTC+8)
2628tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:26192s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:16954s | GPU cuda1:7817/24564MiB:52%;cuda2:4109/24564MiB:100%260820-06:02:14 (UTC+8)
2627tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:26011s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:16773s | GPU cuda1:7817/24564MiB:52%;cuda2:4109/24564MiB:100%260820-05:59:12 (UTC+8)
2626tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:25830s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:16592s | GPU cuda1:7817/24564MiB:52%;cuda2:4107/24564MiB:100%260820-05:56:11 (UTC+8)
2625tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:25649s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:16411s | GPU cuda1:7817/24564MiB:54%;cuda2:4109/24564MiB:100%260820-05:53:10 (UTC+8)
2624tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:25468s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:16230s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-05:50:09 (UTC+8)
2623tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:25287s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:16049s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-05:47:08 (UTC+8)
2622tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:25106s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:15867s | GPU cuda1:7751/24564MiB:53%;cuda2:4107/24564MiB:100%260820-05:44:07 (UTC+8)
2621tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:24925s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:15686s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-05:41:06 (UTC+8)
2620tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:24744s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:15505s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-05:38:05 (UTC+8)
2619tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:24562s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:15324s | GPU cuda1:7751/24564MiB:52%;cuda2:4109/24564MiB:100%260820-05:35:04 (UTC+8)
2618tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:24381s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:15143s | GPU cuda1:7751/24564MiB:51%;cuda2:4107/24564MiB:100%260820-05:32:02 (UTC+8)
2617tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:24199s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:14961s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-05:29:01 (UTC+8)
2616tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:24018s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:14780s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-05:26:00 (UTC+8)
2615tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:23837s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:14599s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-05:22:59 (UTC+8)
2614tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:23656s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:14418s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-05:19:58 (UTC+8)
2613tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:23475s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:14236s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-05:16:56 (UTC+8)
2612tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:23293s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:14055s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-05:13:55 (UTC+8)
2611tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:23112s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:13874s | GPU cuda1:7751/24564MiB:34%;cuda2:4107/24564MiB:100%260820-05:10:53 (UTC+8)
2610tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:22931s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:13692s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-05:07:52 (UTC+8)
2609tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:22750s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:13511s | GPU cuda1:7751/24564MiB:47%;cuda2:4109/24564MiB:100%260820-05:04:51 (UTC+8)
2608tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:22569s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:13330s | GPU cuda1:7751/24564MiB:92%;cuda2:4109/24564MiB:100%260820-05:01:50 (UTC+8)
2607tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:22388s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:13149s | GPU cuda1:7751/24564MiB:41%;cuda2:4107/24564MiB:100%260820-04:58:49 (UTC+8)
2606tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:22207s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:12968s | GPU cuda1:7751/24564MiB:84%;cuda2:4109/24564MiB:100%260820-04:55:48 (UTC+8)
2605tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:22025s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:12787s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-04:52:47 (UTC+8)
2604tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:21844s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:12606s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-04:49:45 (UTC+8)
2603tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:21663s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:12425s | GPU cuda1:7751/24564MiB:54%;cuda2:4107/24564MiB:100%260820-04:46:44 (UTC+8)
2602tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:21482s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:12244s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-04:43:43 (UTC+8)
2601tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:21301s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:12063s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-04:40:42 (UTC+8)
2600tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:21120s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:11882s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-04:37:41 (UTC+8)
2599tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:20939s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:11701s | GPU cuda1:7751/24564MiB:53%;cuda2:4107/24564MiB:100%260820-04:34:40 (UTC+8)
2598tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:20758s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:11519s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-04:31:39 (UTC+8)
2597tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:20577s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:11338s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-04:28:38 (UTC+8)
2596tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:20396s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:11157s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-04:25:37 (UTC+8)
2595tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:20214s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:10976s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-04:22:36 (UTC+8)
2594tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:20033s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:10795s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-04:19:34 (UTC+8)
2593tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:19852s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:10614s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-04:16:33 (UTC+8)
2592tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:19671s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:10433s | GPU cuda1:7751/24564MiB:54%;cuda2:4107/24564MiB:100%260820-04:13:32 (UTC+8)
2591tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:19490s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:10252s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-04:10:31 (UTC+8)
2590tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:19309s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:10070s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-04:07:30 (UTC+8)
2589tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:19127s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:9889s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-04:04:29 (UTC+8)
2588tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:18946s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:9707s | GPU cuda1:7751/24564MiB:51%;cuda2:4107/24564MiB:100%260820-04:01:27 (UTC+8)
2587tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:18764s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:9526s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-03:58:26 (UTC+8)
2586tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:18583s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:9345s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-03:55:24 (UTC+8)
2585tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:18402s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:9164s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-03:52:23 (UTC+8)
2584tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:18221s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:8982s | GPU cuda1:7751/24564MiB:50%;cuda2:4107/24564MiB:100%260820-03:49:22 (UTC+8)
2583tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:18039s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:8801s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-03:46:21 (UTC+8)
2582tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:17858s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:8619s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-03:43:19 (UTC+8)
2581tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:17677s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:8438s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-03:40:18 (UTC+8)
2580tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:17496s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:8257s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-03:37:17 (UTC+8)
2579tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:17315s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:8076s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-03:34:16 (UTC+8)
2578tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:17134s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:7895s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-03:31:15 (UTC+8)
2577tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:16953s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:7714s | GPU cuda1:7751/24564MiB:51%;cuda2:4107/24564MiB:100%260820-03:28:14 (UTC+8)
2576tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:16771s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:7533s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-03:25:13 (UTC+8)
2575tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:16590s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:7351s | GPU cuda1:7751/24564MiB:68%;cuda2:4109/24564MiB:100%260820-03:22:11 (UTC+8)
2574tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:16408s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:7170s | GPU cuda1:7751/24564MiB:52%;cuda2:4109/24564MiB:100%260820-03:19:10 (UTC+8)
2573tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:16227s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:6989s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-03:16:09 (UTC+8)
2572tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:16046s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:6808s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-03:13:08 (UTC+8)
2571tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:15865s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:6626s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-03:10:07 (UTC+8)
2570tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:15684s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:6445s | GPU cuda1:7751/24564MiB:52%;cuda2:4107/24564MiB:100%260820-03:07:05 (UTC+8)
2569tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:15503s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:6264s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-03:04:04 (UTC+8)
2568tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:15322s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:6083s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-03:01:03 (UTC+8)
2567tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:15140s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:5902s | GPU cuda1:7751/24564MiB:49%;cuda2:4109/24564MiB:100%260820-02:58:02 (UTC+8)
2566tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:14959s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:5721s | GPU cuda1:7751/24564MiB:51%;cuda2:4107/24564MiB:100%260820-02:55:00 (UTC+8)
2565tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:14778s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:5540s | GPU cuda1:7751/24564MiB:47%;cuda2:4109/24564MiB:100%260820-02:51:59 (UTC+8)
2564tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:14597s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:5358s | GPU cuda1:7751/24564MiB:52%;cuda2:4109/24564MiB:100%260820-02:48:58 (UTC+8)
2563tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:14416s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:5177s | GPU cuda1:7751/24564MiB:52%;cuda2:4109/24564MiB:100%260820-02:45:57 (UTC+8)
2562tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:14235s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:4996s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-02:42:56 (UTC+8)
2561tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:14053s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:4815s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-02:39:55 (UTC+8)
2560tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:13872s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:4634s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-02:36:53 (UTC+8)
2559tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:13690s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:4452s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-02:33:52 (UTC+8)
2558tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:13509s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:4271s | GPU cuda1:7751/24564MiB:58%;cuda2:4109/24564MiB:100%260820-02:30:51 (UTC+8)
2557tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:13328s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:4090s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-02:27:49 (UTC+8)
2556tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:13147s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:3909s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-02:24:48 (UTC+8)
2555tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:12966s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:3728s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-02:21:47 (UTC+8)
2554tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:12784s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:3546s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-02:18:46 (UTC+8)
2553tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:12603s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:3365s | GPU cuda1:7751/24564MiB:51%;cuda2:4107/24564MiB:100%260820-02:15:45 (UTC+8)
2552tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:12422s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:3184s | GPU cuda1:7751/24564MiB:49%;cuda2:4109/24564MiB:100%260820-02:12:44 (UTC+8)
2551tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:12241s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:3003s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-02:09:43 (UTC+8)
2550tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:12060s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:2822s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-02:06:41 (UTC+8)
2549tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:11878s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:2640s | GPU cuda1:7751/24564MiB:51%;cuda2:4107/24564MiB:100%260820-02:03:40 (UTC+8)
2548tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:11697s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:2459s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-02:00:39 (UTC+8)
2547tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:11516s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:2278s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-01:57:38 (UTC+8)
2546tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:11335s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:2097s | GPU cuda1:7751/24564MiB:53%;cuda2:4109/24564MiB:100%260820-01:54:37 (UTC+8)
2545tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:11154s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:1916s | GPU cuda1:7751/24564MiB:52%;cuda2:4109/24564MiB:100%260820-01:51:35 (UTC+8)
2544tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:10973s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:1735s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-01:48:34 (UTC+8)
2543tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:10792s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:1554s | GPU cuda1:7751/24564MiB:49%;cuda2:4109/24564MiB:100%260820-01:45:33 (UTC+8)
2542tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:10610s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:1372s | GPU cuda1:7751/24564MiB:43%;cuda2:4105/24564MiB:100%260820-01:42:32 (UTC+8)
2541tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:10429s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:1191s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-01:39:31 (UTC+8)
2540tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:10248s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:1010s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-01:36:30 (UTC+8)
2539tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:10067s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:829s | GPU cuda1:7751/24564MiB:52%;cuda2:4109/24564MiB:100%260820-01:33:29 (UTC+8)
2538tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:9886s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:648s | GPU cuda1:7751/24564MiB:54%;cuda2:4107/24564MiB:100%260820-01:30:27 (UTC+8)
2537tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:9705s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:467s | GPU cuda1:7751/24564MiB:54%;cuda2:4109/24564MiB:100%260820-01:27:26 (UTC+8)
2536tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:9524s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:286s | GPU cuda1:7751/24564MiB:50%;cuda2:4109/24564MiB:100%260820-01:24:25 (UTC+8)
2535tasks done=3357/12141 running=2 failed=0 skipped=0 pending=8782 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:9343s;cuda2:hcr_bundle:hcr_bundle__llama3_8b__wikics__s42:105s | GPU cuda1:7751/24564MiB:51%;cuda2:4109/24564MiB:100%260820-01:21:24 (UTC+8)
2534tasks done=3341/12141 running=2 failed=0 skipped=0 pending=8798 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:9162s;cuda2:student_oof:student_oof__wikics__s42__gat3:349s | GPU cuda1:7751/24564MiB:53%;cuda2:16685/24564MiB:100%260820-01:18:23 (UTC+8)
2533tasks done=3341/12141 running=2 failed=0 skipped=0 pending=8798 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:8981s;cuda2:student_oof:student_oof__wikics__s42__gat3:168s | GPU cuda1:7751/24564MiB:54%;cuda2:16485/24564MiB:100%260820-01:15:22 (UTC+8)
2532tasks done=3339/12141 running=2 failed=0 skipped=0 pending=8800 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:8800s;cuda2:student_oof:student_oof__wikics__s42__gat2:229s | GPU cuda1:7751/24564MiB:54%;cuda2:14701/24564MiB:100%260820-01:12:21 (UTC+8)
2531tasks done=3339/12141 running=2 failed=0 skipped=0 pending=8800 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:8618s;cuda2:student_oof:student_oof__wikics__s42__gat2:48s | GPU cuda1:7751/24564MiB:53%;cuda2:13139/24564MiB:100%260820-01:09:20 (UTC+8)
2530tasks done=3335/12141 running=2 failed=0 skipped=0 pending=8804 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:8436s;cuda2:student_oof:student_oof__wikics__s42__sage3:66s | GPU cuda1:7751/24564MiB:54%;cuda2:5139/24564MiB:100%260820-01:06:18 (UTC+8)
2529tasks done=3331/12141 running=2 failed=0 skipped=0 pending=8808 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:8255s;cuda2:student_oof:student_oof__wikics__s42__gcn3:67s | GPU cuda1:7751/24564MiB:53%;cuda2:5235/24564MiB:100%260820-01:03:16 (UTC+8)
2528tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:8074s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:1773s | GPU cuda1:7751/24564MiB:54%;cuda2:16693/24564MiB:100%260820-01:00:15 (UTC+8)
2527tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:7893s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:1591s | GPU cuda1:7751/24564MiB:54%;cuda2:16693/24564MiB:100%260820-00:57:14 (UTC+8)
2526tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:7711s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:1410s | GPU cuda1:7751/24564MiB:54%;cuda2:16693/24564MiB:100%260820-00:54:13 (UTC+8)
2525tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:7531s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:1229s | GPU cuda1:7751/24564MiB:53%;cuda2:16693/24564MiB:100%260820-00:51:12 (UTC+8)
2524tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:7349s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:1048s | GPU cuda1:7751/24564MiB:54%;cuda2:16693/24564MiB:100%260820-00:48:11 (UTC+8)
2523tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:7168s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:867s | GPU cuda1:7751/24564MiB:54%;cuda2:16693/24564MiB:100%260820-00:45:10 (UTC+8)
2522tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:6987s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:686s | GPU cuda1:7751/24564MiB:54%;cuda2:16693/24564MiB:100%260820-00:42:09 (UTC+8)
2521tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:6806s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:504s | GPU cuda1:7751/24564MiB:51%;cuda2:16693/24564MiB:100%260820-00:39:07 (UTC+8)
2520tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:6624s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:323s | GPU cuda1:7751/24564MiB:96%;cuda2:16693/24564MiB:100%260820-00:36:06 (UTC+8)
2519tasks done=3328/12141 running=2 failed=0 skipped=0 pending=8811 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:6443s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat3:141s | GPU cuda1:7751/24564MiB:53%;cuda2:16693/24564MiB:100%260820-00:33:04 (UTC+8)
2518tasks done=3327/12141 running=2 failed=0 skipped=0 pending=8812 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:6262s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat2:1228s | GPU cuda1:7751/24564MiB:53%;cuda2:13145/24564MiB:100%260820-00:30:03 (UTC+8)
2517tasks done=3327/12141 running=2 failed=0 skipped=0 pending=8812 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:6081s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat2:1046s | GPU cuda1:7751/24564MiB:53%;cuda2:13145/24564MiB:100%260820-00:27:02 (UTC+8)
2516tasks done=3327/12141 running=2 failed=0 skipped=0 pending=8812 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:5900s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat2:865s | GPU cuda1:7751/24564MiB:54%;cuda2:13145/24564MiB:100%260820-00:24:01 (UTC+8)
2515tasks done=3327/12141 running=2 failed=0 skipped=0 pending=8812 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:5719s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat2:684s | GPU cuda1:7751/24564MiB:53%;cuda2:13145/24564MiB:100%260820-00:21:00 (UTC+8)
2514tasks done=3327/12141 running=2 failed=0 skipped=0 pending=8812 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:5537s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat2:503s | GPU cuda1:7751/24564MiB:51%;cuda2:13145/24564MiB:100%260820-00:17:59 (UTC+8)
2513tasks done=3327/12141 running=2 failed=0 skipped=0 pending=8812 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:5356s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat2:322s | GPU cuda1:7751/24564MiB:53%;cuda2:13145/24564MiB:100%260820-00:14:58 (UTC+8)
2512tasks done=3327/12141 running=2 failed=0 skipped=0 pending=8812 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:5175s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat2:141s | GPU cuda1:7751/24564MiB:54%;cuda2:13145/24564MiB:100%260820-00:11:57 (UTC+8)
2511tasks done=3326/12141 running=2 failed=0 skipped=0 pending=8813 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:4992s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat1:691s | GPU cuda1:7751/24564MiB:53%;cuda2:11277/24564MiB:100%260820-00:08:55 (UTC+8)
2510tasks done=3326/12141 running=2 failed=0 skipped=0 pending=8813 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:4811s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat1:510s | GPU cuda1:7751/24564MiB:54%;cuda2:11277/24564MiB:100%260820-00:05:52 (UTC+8)
2509tasks done=3326/12141 running=2 failed=0 skipped=0 pending=8813 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:4630s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat1:329s | GPU cuda1:7751/24564MiB:51%;cuda2:11277/24564MiB:100%260820-00:02:51 (UTC+8)
2508tasks done=3326/12141 running=2 failed=0 skipped=0 pending=8813 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:4449s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gat1:148s | GPU cuda1:7751/24564MiB:54%;cuda2:11277/24564MiB:100%260819-23:59:50 (UTC+8)
2507tasks done=3325/12141 running=2 failed=0 skipped=0 pending=8814 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:4267s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__sage3:499s | GPU cuda1:7751/24564MiB:54%;cuda2:5025/24564MiB:100%260819-23:56:49 (UTC+8)
2506tasks done=3325/12141 running=2 failed=0 skipped=0 pending=8814 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:4086s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__sage3:318s | GPU cuda1:7751/24564MiB:52%;cuda2:5025/24564MiB:100%260819-23:53:48 (UTC+8)
2505tasks done=3325/12141 running=2 failed=0 skipped=0 pending=8814 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:3905s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__sage3:136s | GPU cuda1:7751/24564MiB:53%;cuda2:5025/24564MiB:100%260819-23:50:46 (UTC+8)
2504tasks done=3324/12141 running=2 failed=0 skipped=0 pending=8815 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:3724s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__sage2:387s | GPU cuda1:7751/24564MiB:53%;cuda2:5019/24564MiB:100%260819-23:47:45 (UTC+8)
2503tasks done=3324/12141 running=2 failed=0 skipped=0 pending=8815 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:3543s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__sage2:206s | GPU cuda1:7751/24564MiB:53%;cuda2:5019/24564MiB:100%260819-23:44:44 (UTC+8)
2502tasks done=3324/12141 running=2 failed=0 skipped=0 pending=8815 | results done=2090/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:3360s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__sage2:24s | GPU cuda1:7751/24564MiB:50%;cuda2:5019/24564MiB:100%260819-23:41:43 (UTC+8)
2501tasks done=3322/12141 running=2 failed=0 skipped=0 pending=8817 | results done=2089/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:3179s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__sage1:264s | GPU cuda1:7751/24564MiB:53%;cuda2:4635/24564MiB:100%260819-23:38:40 (UTC+8)
2500tasks done=3322/12141 running=2 failed=0 skipped=0 pending=8817 | results done=2089/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:2997s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__sage1:82s | GPU cuda1:7751/24564MiB:54%;cuda2:4635/24564MiB:100%260819-23:35:39 (UTC+8)
2499tasks done=3320/12141 running=2 failed=0 skipped=0 pending=8819 | results done=2088/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:2815s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gcn3:634s | GPU cuda1:7751/24564MiB:87%;cuda2:5221/24564MiB:100%260819-23:32:37 (UTC+8)
2498tasks done=3320/12141 running=2 failed=0 skipped=0 pending=8819 | results done=2088/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:2634s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gcn3:453s | GPU cuda1:7751/24564MiB:54%;cuda2:5221/24564MiB:100%260819-23:29:36 (UTC+8)
2497tasks done=3320/12141 running=2 failed=0 skipped=0 pending=8819 | results done=2088/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:2453s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gcn3:272s | GPU cuda1:7751/24564MiB:54%;cuda2:5221/24564MiB:100%260819-23:26:35 (UTC+8)
2496tasks done=3320/12141 running=2 failed=0 skipped=0 pending=8819 | results done=2088/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:2272s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gcn3:90s | GPU cuda1:7751/24564MiB:51%;cuda2:5221/24564MiB:100%260819-23:23:33 (UTC+8)
2495tasks done=3318/12141 running=2 failed=0 skipped=0 pending=8821 | results done=2087/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:2090s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gcn2:469s | GPU cuda1:7751/24564MiB:53%;cuda2:5137/24564MiB:100%260819-23:20:32 (UTC+8)
2494tasks done=3318/12141 running=2 failed=0 skipped=0 pending=8821 | results done=2087/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:1909s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gcn2:288s | GPU cuda1:7751/24564MiB:51%;cuda2:5137/24564MiB:100%260819-23:17:30 (UTC+8)
2493tasks done=3318/12141 running=2 failed=0 skipped=0 pending=8821 | results done=2087/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:1728s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gcn2:107s | GPU cuda1:7751/24564MiB:54%;cuda2:5137/24564MiB:100%260819-23:14:29 (UTC+8)
2492tasks done=3316/12141 running=2 failed=0 skipped=0 pending=8823 | results done=2086/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:1547s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gcn1:334s | GPU cuda1:7751/24564MiB:54%;cuda2:5093/24564MiB:100%260819-23:11:28 (UTC+8)
2491tasks done=3316/12141 running=2 failed=0 skipped=0 pending=8823 | results done=2086/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:1366s;cuda2:utility_predictions:utility__llama3_8b__wikics__s42__gcn1:153s | GPU cuda1:7751/24564MiB:54%;cuda2:5093/24564MiB:100%260819-23:08:27 (UTC+8)
2490tasks done=3314/12141 running=2 failed=0 skipped=0 pending=8825 | results done=2084/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:1185s;cuda2:task_only:taskonly__wikics__s42__sage1:17s | GPU cuda1:7751/24564MiB:53%;cuda2:4633/24564MiB:100%260819-23:05:26 (UTC+8)
2489tasks done=3310/12141 running=2 failed=0 skipped=0 pending=8829 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:1003s;cuda2:trace_analysis:trace_analysis__llama3_8b__wikics__s42:85s | GPU cuda1:7751/24564MiB:50%;cuda2:5215/24564MiB:100%260819-23:02:25 (UTC+8)
2488tasks done=3308/12141 running=2 failed=0 skipped=0 pending=8831 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:822s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:54863s | GPU cuda1:7751/24564MiB:53%;cuda2:11401/24564MiB:100%260819-22:59:24 (UTC+8)
2487tasks done=3308/12141 running=2 failed=0 skipped=0 pending=8831 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:641s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:54682s | GPU cuda1:7751/24564MiB:54%;cuda2:11401/24564MiB:100%260819-22:56:22 (UTC+8)
2486tasks done=3308/12141 running=2 failed=0 skipped=0 pending=8831 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:460s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:54501s | GPU cuda1:7751/24564MiB:53%;cuda2:11401/24564MiB:100%260819-22:53:21 (UTC+8)
2485tasks done=3308/12141 running=2 failed=0 skipped=0 pending=8831 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:279s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:54320s | GPU cuda1:7751/24564MiB:53%;cuda2:11401/24564MiB:100%260819-22:50:20 (UTC+8)
2484tasks done=3308/12141 running=2 failed=0 skipped=0 pending=8831 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s888:98s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:54138s | GPU cuda1:7751/24564MiB:53%;cuda2:11401/24564MiB:100%260819-22:47:19 (UTC+8)
2483tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:2253s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:53956s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-22:44:17 (UTC+8)
2482tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:2072s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:53775s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-22:41:15 (UTC+8)
2481tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:1891s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:53594s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-22:38:14 (UTC+8)
2480tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:1710s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:53413s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-22:35:13 (UTC+8)
2479tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:1529s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:53232s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-22:32:12 (UTC+8)
2478tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:1347s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:53050s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-22:29:11 (UTC+8)
2477tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:1166s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:52869s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-22:26:10 (UTC+8)
2476tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:985s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:52688s | GPU cuda1:7851/24564MiB:0%;cuda2:11401/24564MiB:100%260819-22:23:09 (UTC+8)
2475tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:804s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:52507s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-22:20:08 (UTC+8)
2474tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:623s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:52326s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-22:17:07 (UTC+8)
2473tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:440s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:52143s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-22:14:06 (UTC+8)
2472tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:259s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:51962s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-22:11:02 (UTC+8)
2471tasks done=3306/12141 running=2 failed=0 skipped=0 pending=8833 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:78s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:51781s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-22:08:01 (UTC+8)
2470tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:6430s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:51600s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-22:05:00 (UTC+8)
2469tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:6249s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:51419s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-22:01:59 (UTC+8)
2468tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:6068s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:51237s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:58:58 (UTC+8)
2467tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:5887s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:51056s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:55:57 (UTC+8)
2466tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:5706s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:50875s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:52:56 (UTC+8)
2465tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:5525s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:50694s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:49:55 (UTC+8)
2464tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:5344s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:50513s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:46:54 (UTC+8)
2463tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:5163s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:50332s | GPU cuda1:11839/24564MiB:99%;cuda2:11401/24564MiB:100%260819-21:43:53 (UTC+8)
2462tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:4982s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:50151s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:40:52 (UTC+8)
2461tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:4801s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:49970s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:37:51 (UTC+8)
2460tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:4620s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:49789s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:34:50 (UTC+8)
2459tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:4439s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:49608s | GPU cuda1:11839/24564MiB:99%;cuda2:11401/24564MiB:100%260819-21:31:49 (UTC+8)
2458tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:4258s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:49427s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:28:48 (UTC+8)
2457tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:4076s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:49245s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:25:47 (UTC+8)
2456tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:3892s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:49062s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:22:45 (UTC+8)
2455tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:3711s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:48881s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:19:41 (UTC+8)
2454tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:3530s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:48700s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:16:40 (UTC+8)
2453tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:3349s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:48519s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:13:39 (UTC+8)
2452tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:3168s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:48338s | GPU cuda1:11839/24564MiB:99%;cuda2:11401/24564MiB:100%260819-21:10:38 (UTC+8)
2451tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:2987s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:48157s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:07:37 (UTC+8)
2450tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:2806s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:47976s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-21:04:36 (UTC+8)
2449tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:2625s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:47795s | GPU cuda1:11839/24564MiB:99%;cuda2:11401/24564MiB:100%260819-21:01:35 (UTC+8)
2448tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:2443s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:47613s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:58:34 (UTC+8)
2447tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:2262s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:47432s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:55:33 (UTC+8)
2446tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:2081s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:47251s | GPU cuda1:11839/24564MiB:99%;cuda2:11401/24564MiB:100%260819-20:52:31 (UTC+8)
2445tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:1900s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:47070s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:49:30 (UTC+8)
2444tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:1719s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:46889s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:46:29 (UTC+8)
2443tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:1537s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:46707s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:43:28 (UTC+8)
2442tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:1356s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:46526s | GPU cuda1:11839/24564MiB:99%;cuda2:11401/24564MiB:100%260819-20:40:26 (UTC+8)
2441tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:1175s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:46345s | GPU cuda1:11839/24564MiB:98%;cuda2:11401/24564MiB:100%260819-20:37:25 (UTC+8)
2440tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:994s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:46164s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:34:24 (UTC+8)
2439tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:813s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:45983s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:31:23 (UTC+8)
2438tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:632s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:45801s | GPU cuda1:11839/24564MiB:95%;cuda2:11401/24564MiB:100%260819-20:28:22 (UTC+8)
2437tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:451s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:45620s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:25:21 (UTC+8)
2436tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:269s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:45439s | GPU cuda1:11839/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:22:20 (UTC+8)
2435tasks done=3305/12141 running=2 failed=0 skipped=0 pending=8834 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:88s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:45258s | GPU cuda1:401/24564MiB:0%;cuda2:11401/24564MiB:100%260819-20:19:18 (UTC+8)
2434tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:2064s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:45077s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-20:16:17 (UTC+8)
2433tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:1883s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:44896s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-20:13:16 (UTC+8)
2432tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:1702s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:44715s | GPU cuda1:7851/24564MiB:99%;cuda2:11401/24564MiB:100%260819-20:10:15 (UTC+8)
2431tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:1521s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:44534s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:07:14 (UTC+8)
2430tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:1340s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:44353s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-20:04:13 (UTC+8)
2429tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:1159s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:44171s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-20:01:12 (UTC+8)
2428tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:977s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:43990s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:58:11 (UTC+8)
2427tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:796s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:43808s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:55:09 (UTC+8)
2426tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:615s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:43627s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260819-19:52:08 (UTC+8)
2425tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:434s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:43446s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:49:07 (UTC+8)
2424tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:253s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:43265s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-19:46:06 (UTC+8)
2423tasks done=3303/12141 running=2 failed=0 skipped=0 pending=8836 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s888:67s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:43080s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:43:05 (UTC+8)
2422tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:2040s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:42898s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-19:39:59 (UTC+8)
2421tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:1859s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:42717s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:36:57 (UTC+8)
2420tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:1678s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:42536s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:33:57 (UTC+8)
2419tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:1497s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:42354s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-19:30:55 (UTC+8)
2418tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:1315s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:42173s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:27:54 (UTC+8)
2417tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:1134s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:41992s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-19:24:53 (UTC+8)
2416tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:953s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:41811s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:21:52 (UTC+8)
2415tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:772s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:41630s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:18:51 (UTC+8)
2414tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:591s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:41449s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:15:50 (UTC+8)
2413tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:410s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:41268s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-19:12:49 (UTC+8)
2412tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:229s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:41087s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-19:09:48 (UTC+8)
2411tasks done=3302/12141 running=2 failed=0 skipped=0 pending=8837 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s888:48s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:40906s | GPU cuda1:7851/24564MiB:0%;cuda2:11401/24564MiB:100%260819-19:06:47 (UTC+8)
2410tasks done=3300/12141 running=2 failed=0 skipped=0 pending=8839 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s888:1209s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:40725s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-19:03:46 (UTC+8)
2409tasks done=3300/12141 running=2 failed=0 skipped=0 pending=8839 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s888:1028s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:40544s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-19:00:45 (UTC+8)
2408tasks done=3300/12141 running=2 failed=0 skipped=0 pending=8839 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s888:847s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:40363s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260819-18:57:43 (UTC+8)
2407tasks done=3300/12141 running=2 failed=0 skipped=0 pending=8839 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s888:666s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:40182s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-18:54:42 (UTC+8)
2406tasks done=3300/12141 running=2 failed=0 skipped=0 pending=8839 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s888:485s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:40001s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-18:51:41 (UTC+8)
2405tasks done=3300/12141 running=2 failed=0 skipped=0 pending=8839 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s888:304s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:39819s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-18:48:40 (UTC+8)
2404tasks done=3300/12141 running=2 failed=0 skipped=0 pending=8839 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s888:123s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:39638s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-18:45:39 (UTC+8)
2403tasks done=3299/12141 running=2 failed=0 skipped=0 pending=8840 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s888:1281s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:39456s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-18:42:38 (UTC+8)
2402tasks done=3299/12141 running=2 failed=0 skipped=0 pending=8840 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s888:1100s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:39275s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-18:39:35 (UTC+8)
2401tasks done=3299/12141 running=2 failed=0 skipped=0 pending=8840 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s888:919s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:39094s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-18:36:34 (UTC+8)
2400tasks done=3299/12141 running=2 failed=0 skipped=0 pending=8840 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s888:738s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:38913s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260819-18:33:33 (UTC+8)
2399tasks done=3299/12141 running=2 failed=0 skipped=0 pending=8840 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s888:557s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:38732s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-18:30:32 (UTC+8)
2398tasks done=3299/12141 running=2 failed=0 skipped=0 pending=8840 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s888:376s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:38551s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-18:27:31 (UTC+8)
2397tasks done=3299/12141 running=2 failed=0 skipped=0 pending=8840 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s888:195s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:38370s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-18:24:30 (UTC+8)
2396tasks done=3299/12141 running=2 failed=0 skipped=0 pending=8840 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s888:13s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:38188s | GPU cuda1:7669/24564MiB:20%;cuda2:11401/24564MiB:100%260819-18:21:29 (UTC+8)
2395tasks done=3297/12141 running=2 failed=0 skipped=0 pending=8842 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s888:754s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:38007s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260819-18:18:27 (UTC+8)
2394tasks done=3297/12141 running=2 failed=0 skipped=0 pending=8842 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s888:572s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:37824s | GPU cuda1:7851/24564MiB:95%;cuda2:11401/24564MiB:100%260819-18:15:26 (UTC+8)
2393tasks done=3297/12141 running=2 failed=0 skipped=0 pending=8842 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s888:391s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:37643s | GPU cuda1:7851/24564MiB:0%;cuda2:11401/24564MiB:100%260819-18:12:24 (UTC+8)
2392tasks done=3297/12141 running=2 failed=0 skipped=0 pending=8842 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s888:209s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:37462s | GPU cuda1:7851/24564MiB:95%;cuda2:11401/24564MiB:100%260819-18:09:23 (UTC+8)
2391tasks done=3297/12141 running=2 failed=0 skipped=0 pending=8842 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s888:28s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:37281s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260819-18:06:21 (UTC+8)
2390tasks done=3296/12141 running=2 failed=0 skipped=0 pending=8843 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s888:767s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:37099s | GPU cuda1:7851/24564MiB:95%;cuda2:11401/24564MiB:100%260819-18:03:20 (UTC+8)
2389tasks done=3296/12141 running=2 failed=0 skipped=0 pending=8843 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s888:586s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:36918s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260819-18:00:19 (UTC+8)
2388tasks done=3296/12141 running=2 failed=0 skipped=0 pending=8843 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s888:405s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:36737s | GPU cuda1:7851/24564MiB:94%;cuda2:11401/24564MiB:100%260819-17:57:18 (UTC+8)
2387tasks done=3296/12141 running=2 failed=0 skipped=0 pending=8843 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s888:224s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:36556s | GPU cuda1:7851/24564MiB:95%;cuda2:11401/24564MiB:100%260819-17:54:17 (UTC+8)
2386tasks done=3296/12141 running=2 failed=0 skipped=0 pending=8843 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s888:43s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:36375s | GPU cuda1:7851/24564MiB:95%;cuda2:11401/24564MiB:100%260819-17:51:16 (UTC+8)
2385tasks done=3294/12141 running=2 failed=0 skipped=0 pending=8845 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s888:658s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:36194s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260819-17:48:15 (UTC+8)
2384tasks done=3294/12141 running=2 failed=0 skipped=0 pending=8845 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s888:477s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:36012s | GPU cuda1:7851/24564MiB:91%;cuda2:11401/24564MiB:100%260819-17:45:13 (UTC+8)
2383tasks done=3294/12141 running=2 failed=0 skipped=0 pending=8845 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s888:296s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:35831s | GPU cuda1:7851/24564MiB:95%;cuda2:11401/24564MiB:100%260819-17:42:12 (UTC+8)
2382tasks done=3294/12141 running=2 failed=0 skipped=0 pending=8845 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s888:114s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:35650s | GPU cuda1:7851/24564MiB:89%;cuda2:11401/24564MiB:100%260819-17:39:11 (UTC+8)
2381tasks done=3293/12141 running=2 failed=0 skipped=0 pending=8846 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s888:731s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:35469s | GPU cuda1:7851/24564MiB:96%;cuda2:11401/24564MiB:100%260819-17:36:09 (UTC+8)
2380tasks done=3293/12141 running=2 failed=0 skipped=0 pending=8846 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s888:550s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:35288s | GPU cuda1:7851/24564MiB:98%;cuda2:11401/24564MiB:100%260819-17:33:08 (UTC+8)
2379tasks done=3293/12141 running=2 failed=0 skipped=0 pending=8846 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s888:368s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:35106s | GPU cuda1:7851/24564MiB:95%;cuda2:11401/24564MiB:100%260819-17:30:07 (UTC+8)
2378tasks done=3293/12141 running=2 failed=0 skipped=0 pending=8846 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s888:186s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:34924s | GPU cuda1:7851/24564MiB:95%;cuda2:11401/24564MiB:100%260819-17:27:05 (UTC+8)
2377tasks done=3293/12141 running=2 failed=0 skipped=0 pending=8846 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s888:4s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:34743s | GPU cuda1:401/24564MiB:0%;cuda2:11401/24564MiB:100%260819-17:24:03 (UTC+8)
2376tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:4354s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:34562s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-17:21:02 (UTC+8)
2375tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:4172s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:34381s | GPU cuda1:7851/24564MiB:0%;cuda2:11401/24564MiB:100%260819-17:18:01 (UTC+8)
2374tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:3991s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:34199s | GPU cuda1:7851/24564MiB:100%;cuda2:11401/24564MiB:100%260819-17:15:00 (UTC+8)
2373tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:3810s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:34018s | GPU cuda1:7851/24564MiB:0%;cuda2:11401/24564MiB:100%260819-17:11:58 (UTC+8)
2372tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:3629s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:33837s | GPU cuda1:7851/24564MiB:0%;cuda2:11401/24564MiB:100%260819-17:08:57 (UTC+8)
2371tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:3447s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:33655s | GPU cuda1:7851/24564MiB:97%;cuda2:11401/24564MiB:100%260819-17:05:56 (UTC+8)
2370tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:3266s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:33474s | GPU cuda1:7851/24564MiB:0%;cuda2:11401/24564MiB:100%260819-17:02:55 (UTC+8)
2369tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:3084s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:33292s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:59:54 (UTC+8)
2368tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:2903s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:33111s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-16:56:51 (UTC+8)
2367tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:2722s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:32930s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-16:53:50 (UTC+8)
2366tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:2541s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:32749s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:50:49 (UTC+8)
2365tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:2360s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:32568s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:47:48 (UTC+8)
2364tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:2178s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:32386s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-16:44:47 (UTC+8)
2363tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:1997s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:32205s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:41:45 (UTC+8)
2362tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:1816s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:32024s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:38:44 (UTC+8)
2361tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:1635s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:31843s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-16:35:43 (UTC+8)
2360tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:1453s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:31661s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-16:32:42 (UTC+8)
2359tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:1271s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:31479s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:29:40 (UTC+8)
2358tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:1090s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:31298s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-16:26:39 (UTC+8)
2357tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:909s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:31117s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-16:23:38 (UTC+8)
2356tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:727s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:30935s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:20:36 (UTC+8)
2355tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:546s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:30754s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-16:17:35 (UTC+8)
2354tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:364s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:30572s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-16:14:34 (UTC+8)
2353tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:183s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:30391s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:11:31 (UTC+8)
2352tasks done=3291/12141 running=2 failed=0 skipped=0 pending=8848 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s888:2s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:30210s | GPU cuda1:401/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:08:30 (UTC+8)
2351tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:4325s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:30028s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-16:05:29 (UTC+8)
2350tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:4144s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:29848s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-16:02:28 (UTC+8)
2349tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:3962s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:29666s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-15:59:27 (UTC+8)
2348tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:3781s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:29485s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-15:56:25 (UTC+8)
2347tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:3600s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:29304s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-15:53:24 (UTC+8)
2346tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:3419s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:29123s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-15:50:23 (UTC+8)
2345tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:3237s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:28941s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-15:47:22 (UTC+8)
2344tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:3056s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:28760s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-15:44:20 (UTC+8)
2343tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:2875s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:28579s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-15:41:19 (UTC+8)
2342tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:2694s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:28397s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-15:38:18 (UTC+8)
2341tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:2512s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:28216s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-15:35:17 (UTC+8)
2340tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:2330s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:28034s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-15:32:16 (UTC+8)
2339tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:2149s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:27853s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-15:29:14 (UTC+8)
2338tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:1966s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:27670s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-15:26:13 (UTC+8)
2337tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:1785s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:27488s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-15:23:09 (UTC+8)
2336tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:1603s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:27307s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-15:20:08 (UTC+8)
2335tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:1420s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:27124s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-15:17:07 (UTC+8)
2334tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:1239s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:26943s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-15:14:04 (UTC+8)
2333tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:1058s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:26762s | GPU cuda1:7851/24564MiB:94%;cuda2:11335/24564MiB:100%260819-15:11:03 (UTC+8)
2332tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:877s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:26581s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-15:08:01 (UTC+8)
2331tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:695s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:26399s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-15:05:00 (UTC+8)
2330tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:514s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:26218s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-15:01:58 (UTC+8)
2329tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:332s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:26036s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:58:57 (UTC+8)
2328tasks done=3290/12141 running=2 failed=0 skipped=0 pending=8849 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s888:151s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:25855s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-14:55:56 (UTC+8)
2327tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:3673s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:25674s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-14:52:55 (UTC+8)
2326tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:3492s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:25493s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:49:54 (UTC+8)
2325tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:3311s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:25312s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-14:46:53 (UTC+8)
2324tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:3130s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:25131s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:43:51 (UTC+8)
2323tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:2949s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:24950s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-14:40:50 (UTC+8)
2322tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:2768s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:24769s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:37:49 (UTC+8)
2321tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:2586s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:24587s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-14:34:48 (UTC+8)
2320tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:2400s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:24401s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:31:46 (UTC+8)
2319tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:2219s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:24220s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-14:28:41 (UTC+8)
2318tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:2038s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:24039s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:25:40 (UTC+8)
2317tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:1857s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:23858s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:22:39 (UTC+8)
2316tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:1676s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:23677s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-14:19:38 (UTC+8)
2315tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:1495s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:23496s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:16:36 (UTC+8)
2314tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:1313s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:23314s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-14:13:35 (UTC+8)
2313tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:1132s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:23133s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:10:34 (UTC+8)
2312tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:951s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:22952s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-14:07:33 (UTC+8)
2311tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:770s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:22771s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-14:04:31 (UTC+8)
2310tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:589s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:22590s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-14:01:30 (UTC+8)
2309tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:408s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:22409s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:58:29 (UTC+8)
2308tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:227s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:22228s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-13:55:28 (UTC+8)
2307tasks done=3288/12141 running=2 failed=0 skipped=0 pending=8851 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s888:43s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:22044s | GPU cuda1:7835/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:52:27 (UTC+8)
2306tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:3548s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:21863s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:49:23 (UTC+8)
2305tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:3367s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:21682s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:46:22 (UTC+8)
2304tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:3185s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:21500s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-13:43:21 (UTC+8)
2303tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:3004s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:21319s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:40:20 (UTC+8)
2302tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:2823s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:21138s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:37:19 (UTC+8)
2301tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:2642s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:20957s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:34:17 (UTC+8)
2300tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:2461s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:20776s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:31:16 (UTC+8)
2299tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:2280s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:20595s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:28:15 (UTC+8)
2298tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:2098s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:20413s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-13:25:14 (UTC+8)
2297tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:1917s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:20232s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:22:12 (UTC+8)
2296tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:1736s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:20051s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-13:19:11 (UTC+8)
2295tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:1555s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:19870s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:16:10 (UTC+8)
2294tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:1374s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:19689s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:13:10 (UTC+8)
2293tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:1188s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:19503s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:10:09 (UTC+8)
2292tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:1006s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:19321s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:07:03 (UTC+8)
2291tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:825s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:19140s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-13:04:00 (UTC+8)
2290tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:644s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:18959s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-13:00:59 (UTC+8)
2289tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:463s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:18778s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-12:57:58 (UTC+8)
2288tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:282s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:18597s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:54:57 (UTC+8)
2287tasks done=3287/12141 running=2 failed=0 skipped=0 pending=8852 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s888:100s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:18415s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-12:51:56 (UTC+8)
2286tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:3114s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:18234s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:48:54 (UTC+8)
2285tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:2933s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:18053s | GPU cuda1:7851/24564MiB:98%;cuda2:11335/24564MiB:100%260819-12:45:53 (UTC+8)
2284tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:2752s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:17872s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-12:42:52 (UTC+8)
2283tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:2571s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:17691s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:39:51 (UTC+8)
2282tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:2390s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:17510s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:36:50 (UTC+8)
2281tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:2209s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:17329s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:33:49 (UTC+8)
2280tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:2028s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:17147s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:30:48 (UTC+8)
2279tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:1847s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:16966s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-12:27:47 (UTC+8)
2278tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:1666s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:16785s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:24:46 (UTC+8)
2277tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:1484s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:16604s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:21:45 (UTC+8)
2276tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:1303s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:16423s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-12:18:44 (UTC+8)
2275tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:1121s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:16241s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:15:43 (UTC+8)
2274tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:940s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:16060s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:12:40 (UTC+8)
2273tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:758s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:15878s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:09:39 (UTC+8)
2272tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:577s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:15697s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-12:06:37 (UTC+8)
2271tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:396s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:15516s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-12:03:36 (UTC+8)
2270tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:215s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:15334s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-12:00:35 (UTC+8)
2269tasks done=3285/12141 running=2 failed=0 skipped=0 pending=8854 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s888:33s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:15153s | GPU cuda1:7835/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:57:34 (UTC+8)
2268tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:3075s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:14972s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:54:32 (UTC+8)
2267tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:2894s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:14790s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:51:31 (UTC+8)
2266tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:2713s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:14609s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-11:48:29 (UTC+8)
2265tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:2532s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:14428s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-11:45:28 (UTC+8)
2264tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:2349s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:14245s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:42:27 (UTC+8)
2263tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:2168s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:14064s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:39:24 (UTC+8)
2262tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:1987s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:13883s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:36:23 (UTC+8)
2261tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:1806s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:13702s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-11:33:22 (UTC+8)
2260tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:1625s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:13521s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:30:21 (UTC+8)
2259tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:1443s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:13339s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:27:20 (UTC+8)
2258tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:1261s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:13157s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:24:19 (UTC+8)
2257tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:1080s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:12976s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-11:21:16 (UTC+8)
2256tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:899s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:12795s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-11:18:15 (UTC+8)
2255tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:718s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:12614s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:15:14 (UTC+8)
2254tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:537s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:12433s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:12:13 (UTC+8)
2253tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:356s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:12252s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:09:12 (UTC+8)
2252tasks done=3284/12141 running=2 failed=0 skipped=0 pending=8855 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s888:175s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:12071s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-11:06:11 (UTC+8)
2251tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:3030s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:11890s | GPU cuda1:7851/24564MiB:98%;cuda2:11335/24564MiB:100%260819-11:03:10 (UTC+8)
2250tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:2849s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:11709s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-11:00:09 (UTC+8)
2249tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:2668s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:11528s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:57:08 (UTC+8)
2248tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:2487s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:11347s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:54:07 (UTC+8)
2247tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:2305s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:11165s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:51:06 (UTC+8)
2246tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:2124s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:10985s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:48:05 (UTC+8)
2245tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:1943s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:10803s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:45:04 (UTC+8)
2244tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:1761s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:10622s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:42:02 (UTC+8)
2243tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:1580s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:10440s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:39:01 (UTC+8)
2242tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:1399s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:10259s | GPU cuda1:7851/24564MiB:98%;cuda2:11335/24564MiB:100%260819-10:36:00 (UTC+8)
2241tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:1218s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:10078s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:32:59 (UTC+8)
2240tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:1036s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:9897s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:29:58 (UTC+8)
2239tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:854s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:9715s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:26:56 (UTC+8)
2238tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:673s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:9534s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:23:54 (UTC+8)
2237tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:492s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:9353s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:20:53 (UTC+8)
2236tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:311s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:9172s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:17:52 (UTC+8)
2235tasks done=3282/12141 running=2 failed=0 skipped=0 pending=8857 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s888:130s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:8991s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:14:51 (UTC+8)
2234tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:3049s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:8809s | GPU cuda1:7851/24564MiB:74%;cuda2:11335/24564MiB:100%260819-10:11:50 (UTC+8)
2233tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:2868s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:8628s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:08:49 (UTC+8)
2232tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:2687s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:8447s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-10:05:48 (UTC+8)
2231tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:2505s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:8266s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-10:02:46 (UTC+8)
2230tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:2324s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:8085s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:59:45 (UTC+8)
2229tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:2143s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:7903s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:56:44 (UTC+8)
2228tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:1962s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:7722s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-09:53:43 (UTC+8)
2227tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:1781s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:7541s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:50:42 (UTC+8)
2226tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:1599s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:7360s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-09:47:41 (UTC+8)
2225tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:1418s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:7179s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:44:39 (UTC+8)
2224tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:1237s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:6998s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-09:41:38 (UTC+8)
2223tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:1056s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:6817s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:38:37 (UTC+8)
2222tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:875s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:6636s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:35:36 (UTC+8)
2221tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:694s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:6455s | GPU cuda1:7851/24564MiB:98%;cuda2:11335/24564MiB:100%260819-09:32:35 (UTC+8)
2220tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:513s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:6274s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:29:34 (UTC+8)
2219tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:332s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:6092s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:26:33 (UTC+8)
2218tasks done=3281/12141 running=2 failed=0 skipped=0 pending=8858 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s888:151s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:5911s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:23:32 (UTC+8)
2217tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:2325s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:5730s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-09:20:31 (UTC+8)
2216tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:2144s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:5549s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-09:17:30 (UTC+8)
2215tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:1963s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:5368s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-09:14:29 (UTC+8)
2214tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:1782s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:5187s | GPU cuda1:7851/24564MiB:0%;cuda2:11335/24564MiB:100%260819-09:11:27 (UTC+8)
2213tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:1601s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:5006s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-09:08:26 (UTC+8)
2212tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:1420s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:4825s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-09:05:25 (UTC+8)
2211tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:1239s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:4644s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-09:02:24 (UTC+8)
2210tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:1057s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:4462s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:59:23 (UTC+8)
2209tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:876s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:4281s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:56:22 (UTC+8)
2208tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:695s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:4100s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:53:21 (UTC+8)
2207tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:514s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:3919s | GPU cuda1:7851/24564MiB:95%;cuda2:11335/24564MiB:100%260819-08:50:20 (UTC+8)
2206tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:333s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:3738s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-08:47:19 (UTC+8)
2205tasks done=3279/12141 running=2 failed=0 skipped=0 pending=8860 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s888:151s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:3556s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:44:18 (UTC+8)
2204tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:2322s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:3375s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-08:41:16 (UTC+8)
2203tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:2141s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:3194s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-08:38:15 (UTC+8)
2202tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:1960s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:3013s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:35:14 (UTC+8)
2201tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:1779s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:2832s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-08:32:12 (UTC+8)
2200tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:1597s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:2651s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-08:29:11 (UTC+8)
2199tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:1416s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:2470s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:26:10 (UTC+8)
2198tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:1235s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:2289s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-08:23:09 (UTC+8)
2197tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:1054s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:2107s | GPU cuda1:7851/24564MiB:94%;cuda2:11335/24564MiB:100%260819-08:20:08 (UTC+8)
2196tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:873s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:1926s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:17:07 (UTC+8)
2195tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:691s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:1745s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:14:06 (UTC+8)
2194tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:510s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:1564s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:11:04 (UTC+8)
2193tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:329s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:1383s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:08:03 (UTC+8)
2192tasks done=3278/12141 running=2 failed=0 skipped=0 pending=8861 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s888:148s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:1202s | GPU cuda1:7851/24564MiB:100%;cuda2:11335/24564MiB:100%260819-08:05:02 (UTC+8)
2191tasks done=3276/12141 running=2 failed=0 skipped=0 pending=8863 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s888:1453s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:1021s | GPU cuda1:7851/24564MiB:97%;cuda2:11335/24564MiB:100%260819-08:02:01 (UTC+8)
2190tasks done=3276/12141 running=2 failed=0 skipped=0 pending=8863 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s888:1272s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:840s | GPU cuda1:7851/24564MiB:96%;cuda2:11335/24564MiB:100%260819-07:59:00 (UTC+8)
2189tasks done=3276/12141 running=2 failed=0 skipped=0 pending=8863 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s888:1090s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:658s | GPU cuda1:7851/24564MiB:98%;cuda2:8675/24564MiB:100%260819-07:55:59 (UTC+8)
2188tasks done=3276/12141 running=2 failed=0 skipped=0 pending=8863 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s888:909s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:477s | GPU cuda1:7851/24564MiB:98%;cuda2:8675/24564MiB:100%260819-07:52:57 (UTC+8)
2187tasks done=3276/12141 running=2 failed=0 skipped=0 pending=8863 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s888:728s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:295s | GPU cuda1:7851/24564MiB:96%;cuda2:8675/24564MiB:49%260819-07:49:56 (UTC+8)
2186tasks done=3276/12141 running=2 failed=0 skipped=0 pending=8863 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s888:546s;cuda2:duo_bank:duo_bank__llama3_8b__wikics__s42:114s | GPU cuda1:7851/24564MiB:97%;cuda2:8675/24564MiB:100%260819-07:46:55 (UTC+8)
2185tasks done=3274/12141 running=2 failed=0 skipped=0 pending=8865 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s888:365s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:4099s | GPU cuda1:7851/24564MiB:97%;cuda2:8775/24564MiB:100%260819-07:43:53 (UTC+8)
2184tasks done=3274/12141 running=2 failed=0 skipped=0 pending=8865 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s888:184s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:3918s | GPU cuda1:7851/24564MiB:98%;cuda2:8775/24564MiB:100%260819-07:40:52 (UTC+8)
2183tasks done=3274/12141 running=2 failed=0 skipped=0 pending=8865 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s888:3s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:3737s | GPU cuda1:401/24564MiB:0%;cuda2:8775/24564MiB:100%260819-07:37:51 (UTC+8)
2182tasks done=3273/12141 running=2 failed=0 skipped=0 pending=8866 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s888:1303s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:3556s | GPU cuda1:7851/24564MiB:98%;cuda2:8775/24564MiB:100%260819-07:34:50 (UTC+8)
2181tasks done=3273/12141 running=2 failed=0 skipped=0 pending=8866 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s888:1122s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:3375s | GPU cuda1:7851/24564MiB:98%;cuda2:8775/24564MiB:100%260819-07:31:49 (UTC+8)
2180tasks done=3273/12141 running=2 failed=0 skipped=0 pending=8866 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s888:940s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:3193s | GPU cuda1:7851/24564MiB:98%;cuda2:8775/24564MiB:100%260819-07:28:47 (UTC+8)
2179tasks done=3273/12141 running=2 failed=0 skipped=0 pending=8866 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s888:759s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:3012s | GPU cuda1:7851/24564MiB:97%;cuda2:8775/24564MiB:100%260819-07:25:46 (UTC+8)
2178tasks done=3273/12141 running=2 failed=0 skipped=0 pending=8866 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s888:578s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:2831s | GPU cuda1:7851/24564MiB:98%;cuda2:8775/24564MiB:100%260819-07:22:45 (UTC+8)
2177tasks done=3273/12141 running=2 failed=0 skipped=0 pending=8866 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s888:396s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:2650s | GPU cuda1:7851/24564MiB:98%;cuda2:8775/24564MiB:100%260819-07:19:44 (UTC+8)
2176tasks done=3273/12141 running=2 failed=0 skipped=0 pending=8866 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s888:215s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:2468s | GPU cuda1:7851/24564MiB:98%;cuda2:8775/24564MiB:100%260819-07:16:42 (UTC+8)
2175tasks done=3273/12141 running=2 failed=0 skipped=0 pending=8866 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s888:34s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:2287s | GPU cuda1:7851/24564MiB:98%;cuda2:8775/24564MiB:100%260819-07:13:41 (UTC+8)
2174tasks done=3271/12141 running=2 failed=0 skipped=0 pending=8868 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s888:754s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:2106s | GPU cuda1:7851/24564MiB:96%;cuda2:8631/24564MiB:100%260819-07:10:40 (UTC+8)
2173tasks done=3271/12141 running=2 failed=0 skipped=0 pending=8868 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s888:573s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:1925s | GPU cuda1:7851/24564MiB:98%;cuda2:8631/24564MiB:100%260819-07:07:39 (UTC+8)
2172tasks done=3271/12141 running=2 failed=0 skipped=0 pending=8868 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s888:393s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:1744s | GPU cuda1:7851/24564MiB:98%;cuda2:8631/24564MiB:100%260819-07:04:38 (UTC+8)
2171tasks done=3271/12141 running=2 failed=0 skipped=0 pending=8868 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s888:211s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:1563s | GPU cuda1:7851/24564MiB:97%;cuda2:8631/24564MiB:98%260819-07:01:37 (UTC+8)
2170tasks done=3271/12141 running=2 failed=0 skipped=0 pending=8868 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s888:30s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:1382s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-06:58:36 (UTC+8)
2169tasks done=3270/12141 running=2 failed=0 skipped=0 pending=8869 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s888:752s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:1201s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-06:55:35 (UTC+8)
2168tasks done=3270/12141 running=2 failed=0 skipped=0 pending=8869 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s888:571s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:1020s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260819-06:52:34 (UTC+8)
2167tasks done=3270/12141 running=2 failed=0 skipped=0 pending=8869 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s888:390s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:839s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260819-06:49:33 (UTC+8)
2166tasks done=3270/12141 running=2 failed=0 skipped=0 pending=8869 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s888:208s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:658s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260819-06:46:32 (UTC+8)
2165tasks done=3270/12141 running=2 failed=0 skipped=0 pending=8869 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s888:27s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:476s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260819-06:43:30 (UTC+8)
2164tasks done=3268/12141 running=2 failed=0 skipped=0 pending=8871 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:2174s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:295s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260819-06:40:29 (UTC+8)
2163tasks done=3268/12141 running=2 failed=0 skipped=0 pending=8871 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:1993s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:114s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260819-06:37:28 (UTC+8)
2162tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:1812s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:14779s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-06:34:27 (UTC+8)
2161tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:1631s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:14598s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-06:31:26 (UTC+8)
2160tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:1449s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:14417s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-06:28:25 (UTC+8)
2159tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:1268s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:14235s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-06:25:24 (UTC+8)
2158tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:1087s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:14054s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-06:22:22 (UTC+8)
2157tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:906s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:13873s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-06:19:21 (UTC+8)
2156tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:725s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:13692s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-06:16:20 (UTC+8)
2155tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:544s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:13511s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-06:13:19 (UTC+8)
2154tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:362s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:13330s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-06:10:18 (UTC+8)
2153tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:181s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:13149s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-06:07:16 (UTC+8)
2152tasks done=3267/12141 running=2 failed=0 skipped=0 pending=8872 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s888:0s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:12968s | GPU cuda1:14/24564MiB:0%;cuda2:15423/24564MiB:100%260819-06:04:15 (UTC+8)
2151tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:2145s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:12786s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-06:01:14 (UTC+8)
2150tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:1963s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:12605s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-05:58:13 (UTC+8)
2149tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:1782s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:12423s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-05:55:12 (UTC+8)
2148tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:1600s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:12242s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-05:52:10 (UTC+8)
2147tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:1419s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:12061s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-05:49:09 (UTC+8)
2146tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:1238s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:11880s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-05:46:08 (UTC+8)
2145tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:1057s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:11699s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-05:43:07 (UTC+8)
2144tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:876s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:11517s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-05:40:06 (UTC+8)
2143tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:695s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:11336s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-05:37:04 (UTC+8)
2142tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:514s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:11156s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-05:34:03 (UTC+8)
2141tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:333s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:10974s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-05:31:02 (UTC+8)
2140tasks done=3266/12141 running=2 failed=0 skipped=0 pending=8873 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s888:151s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:10793s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-05:28:01 (UTC+8)
2139tasks done=3264/12141 running=2 failed=0 skipped=0 pending=8875 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s888:1430s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:10612s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-05:24:59 (UTC+8)
2138tasks done=3264/12141 running=2 failed=0 skipped=0 pending=8875 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s888:1249s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:10431s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-05:21:58 (UTC+8)
2137tasks done=3264/12141 running=2 failed=0 skipped=0 pending=8875 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s888:1068s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:10250s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-05:18:57 (UTC+8)
2136tasks done=3264/12141 running=2 failed=0 skipped=0 pending=8875 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s888:887s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:10069s | GPU cuda1:7851/24564MiB:99%;cuda2:15423/24564MiB:100%260819-05:15:56 (UTC+8)
2135tasks done=3264/12141 running=2 failed=0 skipped=0 pending=8875 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s888:705s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:9887s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-05:12:55 (UTC+8)
2134tasks done=3264/12141 running=2 failed=0 skipped=0 pending=8875 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s888:524s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:9706s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-05:09:54 (UTC+8)
2133tasks done=3264/12141 running=2 failed=0 skipped=0 pending=8875 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s888:343s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:9525s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-05:06:53 (UTC+8)
2132tasks done=3264/12141 running=2 failed=0 skipped=0 pending=8875 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s888:162s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:9343s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-05:03:52 (UTC+8)
2131tasks done=3263/12141 running=2 failed=0 skipped=0 pending=8876 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s888:1438s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:9162s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-05:00:50 (UTC+8)
2130tasks done=3263/12141 running=2 failed=0 skipped=0 pending=8876 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s888:1256s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:8980s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:57:49 (UTC+8)
2129tasks done=3263/12141 running=2 failed=0 skipped=0 pending=8876 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s888:1075s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:8799s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-04:54:47 (UTC+8)
2128tasks done=3263/12141 running=2 failed=0 skipped=0 pending=8876 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s888:894s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:8618s | GPU cuda1:7851/24564MiB:98%;cuda2:15423/24564MiB:100%260819-04:51:46 (UTC+8)
2127tasks done=3263/12141 running=2 failed=0 skipped=0 pending=8876 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s888:713s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:8437s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:48:45 (UTC+8)
2126tasks done=3263/12141 running=2 failed=0 skipped=0 pending=8876 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s888:531s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:8256s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:45:44 (UTC+8)
2125tasks done=3263/12141 running=2 failed=0 skipped=0 pending=8876 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s888:349s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:8074s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:42:43 (UTC+8)
2124tasks done=3263/12141 running=2 failed=0 skipped=0 pending=8876 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s888:168s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:7893s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:39:41 (UTC+8)
2123tasks done=3261/12141 running=2 failed=0 skipped=0 pending=8878 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s888:1043s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:7712s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-04:36:40 (UTC+8)
2122tasks done=3261/12141 running=2 failed=0 skipped=0 pending=8878 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s888:862s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:7531s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:33:39 (UTC+8)
2121tasks done=3261/12141 running=2 failed=0 skipped=0 pending=8878 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s888:681s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:7350s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-04:30:37 (UTC+8)
2120tasks done=3261/12141 running=2 failed=0 skipped=0 pending=8878 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s888:499s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:7168s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:27:36 (UTC+8)
2119tasks done=3261/12141 running=2 failed=0 skipped=0 pending=8878 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s888:318s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:6987s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:24:35 (UTC+8)
2118tasks done=3261/12141 running=2 failed=0 skipped=0 pending=8878 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s888:137s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:6806s | GPU cuda1:7851/24564MiB:98%;cuda2:15423/24564MiB:100%260819-04:21:34 (UTC+8)
2117tasks done=3260/12141 running=2 failed=0 skipped=0 pending=8879 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s888:1010s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:6625s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-04:18:32 (UTC+8)
2116tasks done=3260/12141 running=2 failed=0 skipped=0 pending=8879 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s888:829s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:6444s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:15:31 (UTC+8)
2115tasks done=3260/12141 running=2 failed=0 skipped=0 pending=8879 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s888:648s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:6262s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:12:30 (UTC+8)
2114tasks done=3260/12141 running=2 failed=0 skipped=0 pending=8879 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s888:466s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:6081s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:09:29 (UTC+8)
2113tasks done=3260/12141 running=2 failed=0 skipped=0 pending=8879 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s888:285s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:5900s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-04:06:28 (UTC+8)
2112tasks done=3260/12141 running=2 failed=0 skipped=0 pending=8879 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s888:104s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:5719s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-04:03:27 (UTC+8)
2111tasks done=3258/12141 running=2 failed=0 skipped=0 pending=8881 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s888:802s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:5538s | GPU cuda1:7851/24564MiB:98%;cuda2:15423/24564MiB:100%260819-04:00:26 (UTC+8)
2110tasks done=3258/12141 running=2 failed=0 skipped=0 pending=8881 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s888:621s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:5357s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-03:57:25 (UTC+8)
2109tasks done=3258/12141 running=2 failed=0 skipped=0 pending=8881 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s888:440s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:5176s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-03:54:24 (UTC+8)
2108tasks done=3258/12141 running=2 failed=0 skipped=0 pending=8881 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s888:259s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:4995s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-03:51:23 (UTC+8)
2107tasks done=3258/12141 running=2 failed=0 skipped=0 pending=8881 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s888:78s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:4814s | GPU cuda1:7851/24564MiB:98%;cuda2:15423/24564MiB:100%260819-03:48:21 (UTC+8)
2106tasks done=3257/12141 running=2 failed=0 skipped=0 pending=8882 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s888:773s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:4633s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-03:45:20 (UTC+8)
2105tasks done=3257/12141 running=2 failed=0 skipped=0 pending=8882 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s888:592s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:4452s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-03:42:19 (UTC+8)
2104tasks done=3257/12141 running=2 failed=0 skipped=0 pending=8882 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s888:410s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:4270s | GPU cuda1:7851/24564MiB:98%;cuda2:15423/24564MiB:100%260819-03:39:18 (UTC+8)
2103tasks done=3257/12141 running=2 failed=0 skipped=0 pending=8882 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s888:229s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:4089s | GPU cuda1:7851/24564MiB:97%;cuda2:15423/24564MiB:100%260819-03:36:17 (UTC+8)
2102tasks done=3257/12141 running=2 failed=0 skipped=0 pending=8882 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Random-128-a__llama3_8b__wikics__s888:48s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:3908s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-03:33:16 (UTC+8)
2101tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:2194s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:3727s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-03:30:15 (UTC+8)
2100tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:2011s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:3544s | GPU cuda1:7851/24564MiB:87%;cuda2:15423/24564MiB:100%260819-03:27:14 (UTC+8)
2099tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:1830s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:3363s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-03:24:11 (UTC+8)
2098tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:1649s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:3182s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-03:21:10 (UTC+8)
2097tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:1468s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:3001s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-03:18:09 (UTC+8)
2096tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:1287s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:2820s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-03:15:08 (UTC+8)
2095tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:1105s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:2639s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-03:12:07 (UTC+8)
2094tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:924s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:2458s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-03:09:06 (UTC+8)
2093tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:743s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:2277s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-03:06:05 (UTC+8)
2092tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:562s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:2096s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-03:03:04 (UTC+8)
2091tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:381s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:1915s | GPU cuda1:7851/24564MiB:45%;cuda2:15423/24564MiB:100%260819-03:00:02 (UTC+8)
2090tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:200s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:1733s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-02:57:01 (UTC+8)
2089tasks done=3255/12141 running=2 failed=0 skipped=0 pending=8884 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-b__llama3_8b__wikics__s888:19s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:1552s | GPU cuda1:7713/24564MiB:24%;cuda2:15423/24564MiB:100%260819-02:54:00 (UTC+8)
2088tasks done=3254/12141 running=2 failed=0 skipped=0 pending=8885 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:2163s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:1371s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-02:50:59 (UTC+8)
2087tasks done=3254/12141 running=2 failed=0 skipped=0 pending=8885 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:1982s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:1190s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-02:47:58 (UTC+8)
2086tasks done=3254/12141 running=2 failed=0 skipped=0 pending=8885 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:1800s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:1008s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-02:44:57 (UTC+8)
2085tasks done=3254/12141 running=2 failed=0 skipped=0 pending=8885 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:1619s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:827s | GPU cuda1:7851/24564MiB:96%;cuda2:15423/24564MiB:100%260819-02:41:55 (UTC+8)
2084tasks done=3254/12141 running=2 failed=0 skipped=0 pending=8885 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:1437s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:646s | GPU cuda1:7851/24564MiB:99%;cuda2:15423/24564MiB:100%260819-02:38:54 (UTC+8)
2083tasks done=3254/12141 running=2 failed=0 skipped=0 pending=8885 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:1256s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:465s | GPU cuda1:7851/24564MiB:95%;cuda2:15423/24564MiB:100%260819-02:35:53 (UTC+8)
2082tasks done=3254/12141 running=2 failed=0 skipped=0 pending=8885 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:1075s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:284s | GPU cuda1:7851/24564MiB:100%;cuda2:15423/24564MiB:100%260819-02:32:52 (UTC+8)
2081tasks done=3254/12141 running=2 failed=0 skipped=0 pending=8885 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:894s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:102s | GPU cuda1:7851/24564MiB:100%;cuda2:3985/24564MiB:100%260819-02:29:51 (UTC+8)
2080tasks done=3252/12141 running=2 failed=0 skipped=0 pending=8887 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:713s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:4769s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260819-02:26:49 (UTC+8)
2079tasks done=3252/12141 running=2 failed=0 skipped=0 pending=8887 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:532s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:4588s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260819-02:23:48 (UTC+8)
2078tasks done=3252/12141 running=2 failed=0 skipped=0 pending=8887 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:350s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:4407s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-02:20:47 (UTC+8)
2077tasks done=3252/12141 running=2 failed=0 skipped=0 pending=8887 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-1024-a__llama3_8b__wikics__s888:169s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:4226s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260819-02:17:46 (UTC+8)
2076tasks done=3250/12141 running=2 failed=0 skipped=0 pending=8889 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s888:1045s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:4044s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-02:14:45 (UTC+8)
2075tasks done=3250/12141 running=2 failed=0 skipped=0 pending=8889 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s888:863s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:3863s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-02:11:43 (UTC+8)
2074tasks done=3250/12141 running=2 failed=0 skipped=0 pending=8889 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s888:682s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:3682s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260819-02:08:42 (UTC+8)
2073tasks done=3250/12141 running=2 failed=0 skipped=0 pending=8889 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s888:501s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:3501s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-02:05:41 (UTC+8)
2072tasks done=3250/12141 running=2 failed=0 skipped=0 pending=8889 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s888:320s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:3320s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260819-02:02:40 (UTC+8)
2071tasks done=3250/12141 running=2 failed=0 skipped=0 pending=8889 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-b__llama3_8b__wikics__s888:139s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:3139s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:59:39 (UTC+8)
2070tasks done=3249/12141 running=2 failed=0 skipped=0 pending=8890 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s888:1011s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:2958s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260819-01:56:38 (UTC+8)
2069tasks done=3249/12141 running=2 failed=0 skipped=0 pending=8890 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s888:830s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:2777s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:53:37 (UTC+8)
2068tasks done=3249/12141 running=2 failed=0 skipped=0 pending=8890 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s888:649s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:2596s | GPU cuda1:7851/24564MiB:94%;cuda2:11435/24564MiB:100%260819-01:50:36 (UTC+8)
2067tasks done=3249/12141 running=2 failed=0 skipped=0 pending=8890 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s888:468s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:2414s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:47:35 (UTC+8)
2066tasks done=3249/12141 running=2 failed=0 skipped=0 pending=8890 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s888:287s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:2233s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:44:33 (UTC+8)
2065tasks done=3249/12141 running=2 failed=0 skipped=0 pending=8890 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-256-a__llama3_8b__wikics__s888:106s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:2052s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:41:32 (UTC+8)
2064tasks done=3247/12141 running=2 failed=0 skipped=0 pending=8892 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s888:802s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:1871s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260819-01:38:31 (UTC+8)
2063tasks done=3247/12141 running=2 failed=0 skipped=0 pending=8892 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s888:621s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:1689s | GPU cuda1:7851/24564MiB:26%;cuda2:11435/24564MiB:100%260819-01:35:30 (UTC+8)
2062tasks done=3247/12141 running=2 failed=0 skipped=0 pending=8892 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s888:439s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:1508s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260819-01:32:28 (UTC+8)
2061tasks done=3247/12141 running=2 failed=0 skipped=0 pending=8892 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s888:257s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:1326s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:29:27 (UTC+8)
2060tasks done=3247/12141 running=2 failed=0 skipped=0 pending=8892 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-b__llama3_8b__wikics__s888:76s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:1145s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:26:25 (UTC+8)
2059tasks done=3246/12141 running=2 failed=0 skipped=0 pending=8893 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s888:771s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:964s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:23:24 (UTC+8)
2058tasks done=3246/12141 running=2 failed=0 skipped=0 pending=8893 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s888:590s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:783s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260819-01:20:23 (UTC+8)
2057tasks done=3246/12141 running=2 failed=0 skipped=0 pending=8893 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s888:409s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:602s | GPU cuda1:7851/24564MiB:69%;cuda2:11435/24564MiB:100%260819-01:17:21 (UTC+8)
2056tasks done=3246/12141 running=2 failed=0 skipped=0 pending=8893 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s888:228s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:421s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260819-01:14:20 (UTC+8)
2055tasks done=3246/12141 running=2 failed=0 skipped=0 pending=8893 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-128-a__llama3_8b__wikics__s888:47s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:240s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:11:19 (UTC+8)
2054tasks done=3244/12141 running=2 failed=0 skipped=0 pending=8895 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s888:950s;cuda2:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s42:57s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260819-01:08:18 (UTC+8)
2053tasks done=3243/12141 running=2 failed=0 skipped=0 pending=8896 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s888:769s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:4723s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-01:05:16 (UTC+8)
2052tasks done=3243/12141 running=2 failed=0 skipped=0 pending=8896 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s888:588s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:4542s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260819-01:02:15 (UTC+8)
2051tasks done=3243/12141 running=2 failed=0 skipped=0 pending=8896 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s888:407s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:4361s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260819-00:59:14 (UTC+8)
2050tasks done=3243/12141 running=2 failed=0 skipped=0 pending=8896 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s888:226s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:4180s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260819-00:56:13 (UTC+8)
2049tasks done=3243/12141 running=2 failed=0 skipped=0 pending=8896 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-b__llama3_8b__wikics__s888:45s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:3999s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:53:12 (UTC+8)
2048tasks done=3242/12141 running=2 failed=0 skipped=0 pending=8897 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s888:944s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:3817s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:50:11 (UTC+8)
2047tasks done=3242/12141 running=2 failed=0 skipped=0 pending=8897 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s888:763s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:3636s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260819-00:47:09 (UTC+8)
2046tasks done=3242/12141 running=2 failed=0 skipped=0 pending=8897 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s888:582s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:3455s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260819-00:44:08 (UTC+8)
2045tasks done=3242/12141 running=2 failed=0 skipped=0 pending=8897 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s888:401s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:3274s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:41:07 (UTC+8)
2044tasks done=3242/12141 running=2 failed=0 skipped=0 pending=8897 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s888:219s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:3092s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260819-00:38:06 (UTC+8)
2043tasks done=3242/12141 running=2 failed=0 skipped=0 pending=8897 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-256-a__llama3_8b__wikics__s888:37s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:2910s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260819-00:35:04 (UTC+8)
2042tasks done=3240/12141 running=2 failed=0 skipped=0 pending=8899 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s888:1317s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:2728s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:32:02 (UTC+8)
2041tasks done=3240/12141 running=2 failed=0 skipped=0 pending=8899 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s888:1136s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:2547s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:29:00 (UTC+8)
2040tasks done=3240/12141 running=2 failed=0 skipped=0 pending=8899 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s888:955s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:2366s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:25:59 (UTC+8)
2039tasks done=3240/12141 running=2 failed=0 skipped=0 pending=8899 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s888:773s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:2185s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:22:57 (UTC+8)
2038tasks done=3240/12141 running=2 failed=0 skipped=0 pending=8899 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s888:592s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:2003s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:19:57 (UTC+8)
2037tasks done=3240/12141 running=2 failed=0 skipped=0 pending=8899 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s888:411s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:1822s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:16:55 (UTC+8)
2036tasks done=3240/12141 running=2 failed=0 skipped=0 pending=8899 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s888:230s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:1641s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:13:54 (UTC+8)
2035tasks done=3240/12141 running=2 failed=0 skipped=0 pending=8899 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-b__llama3_8b__wikics__s888:49s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:1460s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:10:53 (UTC+8)
2034tasks done=3239/12141 running=2 failed=0 skipped=0 pending=8900 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s888:1326s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:1279s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:07:52 (UTC+8)
2033tasks done=3239/12141 running=2 failed=0 skipped=0 pending=8900 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s888:1145s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:1098s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260819-00:04:51 (UTC+8)
2032tasks done=3239/12141 running=2 failed=0 skipped=0 pending=8900 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s888:964s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:916s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260819-00:01:49 (UTC+8)
2031tasks done=3239/12141 running=2 failed=0 skipped=0 pending=8900 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s888:783s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:735s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-23:58:48 (UTC+8)
2030tasks done=3239/12141 running=2 failed=0 skipped=0 pending=8900 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s888:599s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:552s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-23:55:47 (UTC+8)
2029tasks done=3239/12141 running=2 failed=0 skipped=0 pending=8900 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s888:418s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:371s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-23:52:44 (UTC+8)
2028tasks done=3239/12141 running=2 failed=0 skipped=0 pending=8900 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s888:237s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:190s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-23:49:43 (UTC+8)
2027tasks done=3239/12141 running=2 failed=0 skipped=0 pending=8900 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Full-512-a__llama3_8b__wikics__s888:56s;cuda2:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s42:9s | GPU cuda1:7851/24564MiB:97%;cuda2:11223/24564MiB:100%260818-23:46:41 (UTC+8)
2026tasks done=3236/12141 running=2 failed=0 skipped=0 pending=8903 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Self__llama3_8b__wikics__s888:600s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:2834s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-23:43:41 (UTC+8)
2025tasks done=3236/12141 running=2 failed=0 skipped=0 pending=8903 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Self__llama3_8b__wikics__s888:418s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:2652s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-23:40:40 (UTC+8)
2024tasks done=3236/12141 running=2 failed=0 skipped=0 pending=8903 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Self__llama3_8b__wikics__s888:237s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:2471s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-23:37:38 (UTC+8)
2023tasks done=3236/12141 running=2 failed=0 skipped=0 pending=8903 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Self__llama3_8b__wikics__s888:56s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:2290s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-23:34:37 (UTC+8)
2022tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:3818s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:2109s | GPU cuda1:10479/24564MiB:99%;cuda2:11435/24564MiB:100%260818-23:31:36 (UTC+8)
2021tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:3637s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:1928s | GPU cuda1:10479/24564MiB:100%;cuda2:11435/24564MiB:100%260818-23:28:35 (UTC+8)
2020tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:3456s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:1747s | GPU cuda1:10479/24564MiB:100%;cuda2:11435/24564MiB:100%260818-23:25:34 (UTC+8)
2019tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:3274s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:1566s | GPU cuda1:10479/24564MiB:99%;cuda2:11435/24564MiB:100%260818-23:22:32 (UTC+8)
2018tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:3093s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:1384s | GPU cuda1:10479/24564MiB:99%;cuda2:11435/24564MiB:100%260818-23:19:31 (UTC+8)
2017tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:2911s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:1203s | GPU cuda1:10479/24564MiB:99%;cuda2:11435/24564MiB:100%260818-23:16:30 (UTC+8)
2016tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:2730s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:1021s | GPU cuda1:10479/24564MiB:100%;cuda2:11435/24564MiB:100%260818-23:13:29 (UTC+8)
2015tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:2549s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:840s | GPU cuda1:10479/24564MiB:99%;cuda2:11435/24564MiB:100%260818-23:10:27 (UTC+8)
2014tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:2368s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:659s | GPU cuda1:10479/24564MiB:99%;cuda2:11435/24564MiB:100%260818-23:07:26 (UTC+8)
2013tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:2186s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:478s | GPU cuda1:10479/24564MiB:100%;cuda2:11435/24564MiB:100%260818-23:04:25 (UTC+8)
2012tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:2004s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:295s | GPU cuda1:10479/24564MiB:99%;cuda2:11435/24564MiB:100%260818-23:01:24 (UTC+8)
2011tasks done=3235/12141 running=2 failed=0 skipped=0 pending=8904 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:1822s;cuda2:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s42:114s | GPU cuda1:10479/24564MiB:99%;cuda2:11435/24564MiB:100%260818-22:58:21 (UTC+8)
2010tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:1641s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:2932s | GPU cuda1:10479/24564MiB:94%;cuda2:11435/24564MiB:100%260818-22:55:20 (UTC+8)
2009tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:1460s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:2752s | GPU cuda1:10477/24564MiB:99%;cuda2:11435/24564MiB:100%260818-22:52:18 (UTC+8)
2008tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:1279s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:2570s | GPU cuda1:10477/24564MiB:100%;cuda2:11435/24564MiB:100%260818-22:49:18 (UTC+8)
2007tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:1098s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:2389s | GPU cuda1:10477/24564MiB:99%;cuda2:11435/24564MiB:100%260818-22:46:16 (UTC+8)
2006tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:916s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:2207s | GPU cuda1:10477/24564MiB:99%;cuda2:11435/24564MiB:100%260818-22:43:15 (UTC+8)
2005tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:734s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:2025s | GPU cuda1:10477/24564MiB:100%;cuda2:11435/24564MiB:100%260818-22:40:13 (UTC+8)
2004tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:552s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:1844s | GPU cuda1:10477/24564MiB:99%;cuda2:11435/24564MiB:100%260818-22:37:11 (UTC+8)
2003tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:369s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:1661s | GPU cuda1:10477/24564MiB:100%;cuda2:11435/24564MiB:100%260818-22:34:10 (UTC+8)
2002tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:188s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:1480s | GPU cuda1:10477/24564MiB:99%;cuda2:11435/24564MiB:100%260818-22:31:07 (UTC+8)
2001tasks done=3234/12141 running=2 failed=0 skipped=0 pending=8905 | results done=2081/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s888:7s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:1299s | GPU cuda1:401/24564MiB:0%;cuda2:11435/24564MiB:100%260818-22:28:06 (UTC+8)
2000tasks done=3231/12141 running=2 failed=0 skipped=0 pending=8908 | results done=2079/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s666__gat3:79s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:1118s | GPU cuda1:13091/24564MiB:97%;cuda2:11435/24564MiB:100%260818-22:25:05 (UTC+8)
1999tasks done=3228/12141 running=2 failed=0 skipped=0 pending=8911 | results done=2076/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s666__gat3:70s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:936s | GPU cuda1:13107/24564MiB:96%;cuda2:11435/24564MiB:100%260818-22:22:04 (UTC+8)
1998tasks done=3226/12141 running=2 failed=0 skipped=0 pending=8913 | results done=2074/8095 failed=0 skipped=0 | RUNNING cuda1:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s666__gat3:103s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:755s | GPU cuda1:13109/24564MiB:94%;cuda2:11435/24564MiB:100%260818-22:19:02 (UTC+8)
1997tasks done=3224/12141 running=2 failed=0 skipped=0 pending=8915 | results done=2072/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s666__gat2:43s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:574s | GPU cuda1:11203/24564MiB:94%;cuda2:11435/24564MiB:100%260818-22:16:01 (UTC+8)
1996tasks done=3221/12141 running=2 failed=0 skipped=0 pending=8918 | results done=2069/8095 failed=0 skipped=0 | RUNNING cuda1:external_downstream:run__LinguGKD__llama3_8b__wikics__s666__gat2:14s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:393s | GPU cuda1:11729/24564MiB:68%;cuda2:11435/24564MiB:100%260818-22:13:00 (UTC+8)
1995tasks done=3218/12141 running=2 failed=0 skipped=0 pending=8921 | results done=2066/8095 failed=0 skipped=0 | RUNNING cuda1:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s666__gat2:10s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:212s | GPU cuda1:9561/24564MiB:94%;cuda2:11435/24564MiB:100%260818-22:09:59 (UTC+8)
1994tasks done=3215/12141 running=2 failed=0 skipped=0 pending=8924 | results done=2063/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s666__gat1:8s;cuda2:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s42:30s | GPU cuda1:7695/24564MiB:92%;cuda2:11435/24564MiB:100%260818-22:06:58 (UTC+8)
1993tasks done=3207/12141 running=2 failed=0 skipped=0 pending=8932 | results done=2057/8095 failed=0 skipped=0 | RUNNING cuda1:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s666__gat1:6s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:1891s | GPU cuda1:7693/24564MiB:92%;cuda2:11435/24564MiB:100%260818-22:03:56 (UTC+8)
1992tasks done=3197/12141 running=2 failed=0 skipped=0 pending=8942 | results done=2047/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T7_TRACEBilevelSoft__llama3_8b__wikics__s666__sage2:16s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:1710s | GPU cuda1:1195/24564MiB:72%;cuda2:11435/24564MiB:100%260818-22:00:55 (UTC+8)
1991tasks done=3185/12141 running=2 failed=0 skipped=0 pending=8954 | results done=2035/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T3_TeacherOptGlobal__llama3_8b__wikics__s666__sage1:8s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:1529s | GPU cuda1:1051/24564MiB:64%;cuda2:11435/24564MiB:100%260818-21:57:54 (UTC+8)
1990tasks done=3177/12141 running=2 failed=0 skipped=0 pending=8962 | results done=2027/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T3_TeacherOptGlobal__llama3_8b__wikics__s666__gcn3:12s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:1348s | GPU cuda1:1637/24564MiB:79%;cuda2:11435/24564MiB:100%260818-21:54:53 (UTC+8)
1989tasks done=3166/12141 running=2 failed=0 skipped=0 pending=8973 | results done=2016/8095 failed=0 skipped=0 | RUNNING cuda1:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s666__gcn2:22s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:1166s | GPU cuda1:1633/24564MiB:78%;cuda2:11435/24564MiB:100%260818-21:51:52 (UTC+8)
1988tasks done=3158/12141 running=2 failed=0 skipped=0 pending=8981 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:oracle_granularity:oracle__llama3_8b__wikics__s666:3s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:985s | GPU cuda1:401/24564MiB:0%;cuda2:11435/24564MiB:100%260818-21:48:50 (UTC+8)
1987tasks done=3156/12141 running=2 failed=0 skipped=0 pending=8983 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:student_oof:student_oof__wikics__s666__gat2:60s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:804s | GPU cuda1:11217/24564MiB:96%;cuda2:11435/24564MiB:100%260818-21:45:49 (UTC+8)
1986tasks done=3151/12141 running=2 failed=0 skipped=0 pending=8988 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:student_oof:student_oof__wikics__s666__gcn3:17s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:622s | GPU cuda1:1631/24564MiB:88%;cuda2:11435/24564MiB:100%260818-21:42:48 (UTC+8)
1985tasks done=3148/12141 running=2 failed=0 skipped=0 pending=8991 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__gat3:496s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:441s | GPU cuda1:13109/24564MiB:96%;cuda2:11435/24564MiB:100%260818-21:39:46 (UTC+8)
1984tasks done=3148/12141 running=2 failed=0 skipped=0 pending=8991 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__gat3:315s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:260s | GPU cuda1:13109/24564MiB:96%;cuda2:11435/24564MiB:100%260818-21:36:45 (UTC+8)
1983tasks done=3148/12141 running=2 failed=0 skipped=0 pending=8991 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__gat3:133s;cuda2:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s42:78s | GPU cuda1:13109/24564MiB:95%;cuda2:11435/24564MiB:100%260818-21:33:44 (UTC+8)
1982tasks done=3146/12141 running=2 failed=0 skipped=0 pending=8993 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__gat2:373s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:1938s | GPU cuda1:9561/24564MiB:93%;cuda2:11435/24564MiB:100%260818-21:30:42 (UTC+8)
1981tasks done=3146/12141 running=2 failed=0 skipped=0 pending=8993 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__gat2:192s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:1757s | GPU cuda1:9561/24564MiB:94%;cuda2:11435/24564MiB:100%260818-21:27:41 (UTC+8)
1980tasks done=3146/12141 running=2 failed=0 skipped=0 pending=8993 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__gat2:11s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:1577s | GPU cuda1:9561/24564MiB:93%;cuda2:11435/24564MiB:100%260818-21:24:40 (UTC+8)
1979tasks done=3145/12141 running=2 failed=0 skipped=0 pending=8994 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__gat1:49s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:1395s | GPU cuda1:7693/24564MiB:93%;cuda2:11435/24564MiB:100%260818-21:21:39 (UTC+8)
1978tasks done=3143/12141 running=2 failed=0 skipped=0 pending=8996 | results done=2009/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__sage2:47s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:1214s | GPU cuda1:1437/24564MiB:70%;cuda2:11435/24564MiB:100%260818-21:18:38 (UTC+8)
1977tasks done=3139/12141 running=2 failed=0 skipped=0 pending=9000 | results done=2007/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__gcn3:114s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:1033s | GPU cuda1:1637/24564MiB:80%;cuda2:11435/24564MiB:100%260818-21:15:37 (UTC+8)
1976tasks done=3137/12141 running=2 failed=0 skipped=0 pending=9002 | results done=2006/8095 failed=0 skipped=0 | RUNNING cuda1:utility_predictions:utility__llama3_8b__wikics__s666__gcn2:52s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:852s | GPU cuda1:1553/24564MiB:77%;cuda2:11435/24564MiB:100%260818-21:12:36 (UTC+8)
1975tasks done=3129/12141 running=2 failed=0 skipped=0 pending=9010 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_analysis:trace_analysis__llama3_8b__wikics__s666:49s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:671s | GPU cuda1:1631/24564MiB:7%;cuda2:11435/24564MiB:100%260818-21:09:35 (UTC+8)
1974tasks done=3127/12141 running=2 failed=0 skipped=0 pending=9012 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:41710s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:490s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-21:06:34 (UTC+8)
1973tasks done=3127/12141 running=2 failed=0 skipped=0 pending=9012 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:41529s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:308s | GPU cuda1:7817/24564MiB:55%;cuda2:11435/24564MiB:100%260818-21:03:33 (UTC+8)
1972tasks done=3127/12141 running=2 failed=0 skipped=0 pending=9012 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:41347s;cuda2:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s42:127s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-21:00:31 (UTC+8)
1971tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:41166s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:1700s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-20:57:30 (UTC+8)
1970tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:40985s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:1519s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-20:54:28 (UTC+8)
1969tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:40803s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:1337s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-20:51:27 (UTC+8)
1968tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:40622s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:1156s | GPU cuda1:7817/24564MiB:86%;cuda2:11435/24564MiB:100%260818-20:48:26 (UTC+8)
1967tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:40441s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:975s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-20:45:24 (UTC+8)
1966tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:40260s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:794s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-20:42:23 (UTC+8)
1965tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:40079s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:613s | GPU cuda1:7817/24564MiB:52%;cuda2:11435/24564MiB:100%260818-20:39:22 (UTC+8)
1964tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:39897s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:430s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-20:36:21 (UTC+8)
1963tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:39715s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:249s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-20:33:19 (UTC+8)
1962tasks done=3125/12141 running=2 failed=0 skipped=0 pending=9014 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:39534s;cuda2:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s42:68s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-20:30:18 (UTC+8)
1961tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:39353s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:1640s | GPU cuda1:7817/24564MiB:52%;cuda2:11435/24564MiB:100%260818-20:27:17 (UTC+8)
1960tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:39172s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:1459s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-20:24:16 (UTC+8)
1959tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:38991s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:1278s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-20:21:15 (UTC+8)
1958tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:38810s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:1097s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-20:18:13 (UTC+8)
1957tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:38629s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:916s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-20:15:12 (UTC+8)
1956tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:38448s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:735s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-20:12:11 (UTC+8)
1955tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:38266s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:554s | GPU cuda1:7817/24564MiB:50%;cuda2:11435/24564MiB:100%260818-20:09:10 (UTC+8)
1954tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:38085s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:372s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-20:06:09 (UTC+8)
1953tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:37904s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:191s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-20:03:07 (UTC+8)
1952tasks done=3124/12141 running=2 failed=0 skipped=0 pending=9015 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:37722s;cuda2:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s42:9s | GPU cuda1:7817/24564MiB:53%;cuda2:11229/24564MiB:100%260818-20:00:06 (UTC+8)
1951tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:37541s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:7257s | GPU cuda1:7817/24564MiB:55%;cuda2:11435/24564MiB:100%260818-19:57:04 (UTC+8)
1950tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:37360s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:7076s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:54:04 (UTC+8)
1949tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:37179s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:6894s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:51:03 (UTC+8)
1948tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:36998s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:6713s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-19:48:01 (UTC+8)
1947tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:36817s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:6532s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:45:00 (UTC+8)
1946tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:36636s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:6351s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:41:59 (UTC+8)
1945tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:36455s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:6170s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:38:58 (UTC+8)
1944tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:36273s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:5989s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-19:35:57 (UTC+8)
1943tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:36092s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:5807s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-19:32:55 (UTC+8)
1942tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:35911s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:5626s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:29:54 (UTC+8)
1941tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:35730s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:5445s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:26:53 (UTC+8)
1940tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:35549s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:5264s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:23:52 (UTC+8)
1939tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:35366s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:5082s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:20:51 (UTC+8)
1938tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:35185s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:4901s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:17:48 (UTC+8)
1937tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:35004s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:4720s | GPU cuda1:7817/24564MiB:65%;cuda2:11435/24564MiB:100%260818-19:14:47 (UTC+8)
1936tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:34823s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:4539s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:11:46 (UTC+8)
1935tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:34642s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:4357s | GPU cuda1:7817/24564MiB:54%;cuda2:11445/24564MiB:100%260818-19:08:45 (UTC+8)
1934tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:34460s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:4176s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-19:05:44 (UTC+8)
1933tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:34279s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:3995s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-19:02:43 (UTC+8)
1932tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:34098s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:3814s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:59:42 (UTC+8)
1931tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:33917s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:3633s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:56:41 (UTC+8)
1930tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:33736s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:3451s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:53:39 (UTC+8)
1929tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:33555s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:3270s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:50:38 (UTC+8)
1928tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:33373s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:3088s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-18:47:37 (UTC+8)
1927tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:33192s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:2907s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:44:35 (UTC+8)
1926tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:33010s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:2726s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:41:34 (UTC+8)
1925tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:32829s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:2545s | GPU cuda1:7817/24564MiB:52%;cuda2:11435/24564MiB:100%260818-18:38:33 (UTC+8)
1924tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:32648s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:2364s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-18:35:31 (UTC+8)
1923tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:32467s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:2183s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:32:30 (UTC+8)
1922tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:32286s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:2002s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:29:29 (UTC+8)
1921tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:32104s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:1819s | GPU cuda1:7817/24564MiB:52%;cuda2:11435/24564MiB:100%260818-18:26:28 (UTC+8)
1920tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:31922s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:1638s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:23:26 (UTC+8)
1919tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:31741s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:1457s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-18:20:25 (UTC+8)
1918tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:31560s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:1276s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-18:17:24 (UTC+8)
1917tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:31379s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:1095s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-18:14:23 (UTC+8)
1916tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:31198s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:913s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-18:11:21 (UTC+8)
1915tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:31017s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:732s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:08:20 (UTC+8)
1914tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:30836s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:551s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:05:19 (UTC+8)
1913tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:30655s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:370s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-18:02:18 (UTC+8)
1912tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:30473s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:189s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-17:59:17 (UTC+8)
1911tasks done=3122/12141 running=2 failed=0 skipped=0 pending=9017 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:30292s;cuda2:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s42:8s | GPU cuda1:7817/24564MiB:54%;cuda2:11217/24564MiB:100%260818-17:56:16 (UTC+8)
1910tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:30111s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:7306s | GPU cuda1:7817/24564MiB:50%;cuda2:11435/24564MiB:100%260818-17:53:15 (UTC+8)
1909tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:29928s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:7123s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-17:50:14 (UTC+8)
1908tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:29746s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:6941s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-17:47:10 (UTC+8)
1907tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:29565s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:6760s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-17:44:09 (UTC+8)
1906tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:29385s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:6579s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-17:41:08 (UTC+8)
1905tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:29203s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:6398s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-17:38:07 (UTC+8)
1904tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:29022s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:6217s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-17:35:06 (UTC+8)
1903tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:28841s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:6036s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-17:32:05 (UTC+8)
1902tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:28660s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:5855s | GPU cuda1:7817/24564MiB:83%;cuda2:11435/24564MiB:100%260818-17:29:04 (UTC+8)
1901tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:28479s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:5673s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-17:26:02 (UTC+8)
1900tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:28298s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:5492s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-17:23:01 (UTC+8)
1899tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:28117s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:5311s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-17:20:00 (UTC+8)
1898tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:27936s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:5130s | GPU cuda1:7817/24564MiB:50%;cuda2:11435/24564MiB:100%260818-17:16:59 (UTC+8)
1897tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:27755s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:4949s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-17:13:58 (UTC+8)
1896tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:27573s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:4768s | GPU cuda1:7817/24564MiB:51%;cuda2:11435/24564MiB:100%260818-17:10:57 (UTC+8)
1895tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:27392s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:4586s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-17:07:56 (UTC+8)
1894tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:27211s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:4405s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-17:04:54 (UTC+8)
1893tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:27030s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:4224s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-17:01:53 (UTC+8)
1892tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:26849s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:4043s | GPU cuda1:7817/24564MiB:55%;cuda2:11435/24564MiB:100%260818-16:58:52 (UTC+8)
1891tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:26667s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:3862s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-16:55:51 (UTC+8)
1890tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:26486s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:3681s | GPU cuda1:7817/24564MiB:53%;cuda2:11435/24564MiB:100%260818-16:52:49 (UTC+8)
1889tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:26305s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:3500s | GPU cuda1:7817/24564MiB:90%;cuda2:11435/24564MiB:100%260818-16:49:48 (UTC+8)
1888tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:26124s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:3319s | GPU cuda1:7817/24564MiB:66%;cuda2:11435/24564MiB:100%260818-16:46:47 (UTC+8)
1887tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:25943s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:3138s | GPU cuda1:7817/24564MiB:50%;cuda2:11435/24564MiB:100%260818-16:43:46 (UTC+8)
1886tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:25761s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:2956s | GPU cuda1:7817/24564MiB:54%;cuda2:11435/24564MiB:100%260818-16:40:45 (UTC+8)
1885tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:25580s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:2775s | GPU cuda1:7751/24564MiB:51%;cuda2:11445/24564MiB:100%260818-16:37:44 (UTC+8)
1884tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:25399s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:2594s | GPU cuda1:7751/24564MiB:54%;cuda2:11445/24564MiB:100%260818-16:34:42 (UTC+8)
1883tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:25218s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:2413s | GPU cuda1:7751/24564MiB:52%;cuda2:11445/24564MiB:100%260818-16:31:41 (UTC+8)
1882tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:25037s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:2232s | GPU cuda1:7751/24564MiB:51%;cuda2:11445/24564MiB:100%260818-16:28:40 (UTC+8)
1881tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:24856s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:2050s | GPU cuda1:7751/24564MiB:51%;cuda2:11445/24564MiB:100%260818-16:25:39 (UTC+8)
1880tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:24674s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:1869s | GPU cuda1:7751/24564MiB:55%;cuda2:11445/24564MiB:100%260818-16:22:38 (UTC+8)
1879tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:24493s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:1688s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-16:19:36 (UTC+8)
1878tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:24312s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:1507s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-16:16:35 (UTC+8)
1877tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:23750s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:945s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-16:07:13 (UTC+8)
1876tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:23569s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:763s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-16:04:12 (UTC+8)
1875tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:23388s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:582s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-16:01:11 (UTC+8)
1874tasks done=3121/12141 running=2 failed=0 skipped=0 pending=9018 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:23016s;cuda2:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s42:211s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-15:54:59 (UTC+8)
1873tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:22459s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:5164s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-15:45:43 (UTC+8)
1872tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:22090s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:4795s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-15:39:34 (UTC+8)
1871tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:21909s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:4614s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-15:36:33 (UTC+8)
1870tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:21728s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:4433s | GPU cuda1:7751/24564MiB:56%;cuda2:11435/24564MiB:100%260818-15:33:32 (UTC+8)
1869tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:21547s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:4251s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-15:30:31 (UTC+8)
1868tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:21366s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:4070s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-15:27:29 (UTC+8)
1867tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:20045s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:2749s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-15:05:28 (UTC+8)
1866tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:19863s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:2568s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-15:02:27 (UTC+8)
1865tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:19682s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:2387s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-14:59:26 (UTC+8)
1864tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:19501s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:2206s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-14:56:25 (UTC+8)
1863tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:19320s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:2024s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-14:53:24 (UTC+8)
1862tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:19139s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:1843s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-14:50:22 (UTC+8)
1861tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:18957s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:1661s | GPU cuda1:7751/24564MiB:95%;cuda2:11435/24564MiB:100%260818-14:47:21 (UTC+8)
1860tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:18776s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:1480s | GPU cuda1:7751/24564MiB:55%;cuda2:11435/24564MiB:100%260818-14:44:19 (UTC+8)
1859tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:18594s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:1298s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-14:41:18 (UTC+8)
1858tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:18412s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:1117s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-14:38:16 (UTC+8)
1857tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:18231s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:935s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-14:35:15 (UTC+8)
1856tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:18050s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:754s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-14:32:13 (UTC+8)
1855tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:17869s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:573s | GPU cuda1:7751/24564MiB:55%;cuda2:11435/24564MiB:100%260818-14:29:12 (UTC+8)
1854tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:17686s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:390s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-14:26:11 (UTC+8)
1853tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:17505s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:209s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-14:23:08 (UTC+8)
1852tasks done=3119/12141 running=2 failed=0 skipped=0 pending=9020 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:17323s;cuda2:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s42:27s | GPU cuda1:7751/24564MiB:54%;cuda2:11419/24564MiB:100%260818-14:20:07 (UTC+8)
1851tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:17142s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:5267s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-14:17:05 (UTC+8)
1850tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:16961s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:5086s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-14:14:04 (UTC+8)
1849tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:16780s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:4905s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-14:11:03 (UTC+8)
1848tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:16598s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:4724s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-14:08:02 (UTC+8)
1847tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:16415s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:4541s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-14:05:01 (UTC+8)
1846tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:16234s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:4360s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-14:01:58 (UTC+8)
1845tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:16053s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:4179s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-13:58:57 (UTC+8)
1844tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:15872s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:3998s | GPU cuda1:7751/24564MiB:49%;cuda2:11435/24564MiB:100%260818-13:55:56 (UTC+8)
1843tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:15691s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:3816s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-13:52:55 (UTC+8)
1842tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:15510s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:3635s | GPU cuda1:7751/24564MiB:100%;cuda2:11435/24564MiB:100%260818-13:49:53 (UTC+8)
1841tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:15329s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:3454s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-13:46:52 (UTC+8)
1840tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:15148s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:3273s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-13:43:51 (UTC+8)
1839tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:14967s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:3092s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-13:40:50 (UTC+8)
1838tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:14784s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:2909s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-13:37:49 (UTC+8)
1837tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:14602s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:2727s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-13:34:46 (UTC+8)
1836tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:14420s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:2546s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-13:31:44 (UTC+8)
1835tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:14239s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:2364s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-13:28:43 (UTC+8)
1834tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:14058s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:2183s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-13:25:41 (UTC+8)
1833tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:13877s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:2002s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-13:22:40 (UTC+8)
1832tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:13696s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:1821s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-13:19:39 (UTC+8)
1831tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:13514s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:1639s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-13:16:38 (UTC+8)
1830tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:13332s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:1458s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-13:13:36 (UTC+8)
1829tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:13151s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:1276s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-13:10:35 (UTC+8)
1828tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:12969s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:1095s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-13:07:33 (UTC+8)
1827tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:12788s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:913s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-13:04:32 (UTC+8)
1826tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:12607s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:732s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-13:01:30 (UTC+8)
1825tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:12425s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:550s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:58:29 (UTC+8)
1824tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:12243s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:369s | GPU cuda1:7751/24564MiB:55%;cuda2:11435/24564MiB:100%260818-12:55:27 (UTC+8)
1823tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:12062s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:188s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-12:52:26 (UTC+8)
1822tasks done=3118/12141 running=2 failed=0 skipped=0 pending=9021 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:11881s;cuda2:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s42:7s | GPU cuda1:7751/24564MiB:51%;cuda2:11209/24564MiB:100%260818-12:49:25 (UTC+8)
1821tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:11700s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:4358s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-12:46:24 (UTC+8)
1820tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:11518s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:4176s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-12:43:23 (UTC+8)
1819tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:11337s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:3995s | GPU cuda1:7751/24564MiB:55%;cuda2:11435/24564MiB:100%260818-12:40:21 (UTC+8)
1818tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:11156s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:3814s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-12:37:20 (UTC+8)
1817tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:10975s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:3633s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:34:19 (UTC+8)
1816tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:10794s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:3452s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:31:18 (UTC+8)
1815tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:10609s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:3266s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:28:16 (UTC+8)
1814tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:10427s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:3085s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:25:11 (UTC+8)
1813tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:10246s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:2903s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-12:22:09 (UTC+8)
1812tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:10065s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:2722s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-12:19:08 (UTC+8)
1811tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:9884s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:2541s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:16:07 (UTC+8)
1810tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:9703s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:2360s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:13:06 (UTC+8)
1809tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:9521s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:2179s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:10:05 (UTC+8)
1808tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:9341s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:1998s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-12:07:04 (UTC+8)
1807tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:9158s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:1816s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:04:03 (UTC+8)
1806tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:8977s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:1634s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-12:01:01 (UTC+8)
1805tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:8796s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:1453s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:57:59 (UTC+8)
1804tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:8615s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:1272s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:54:58 (UTC+8)
1803tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:8434s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:1091s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:51:57 (UTC+8)
1802tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:8252s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:910s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:48:56 (UTC+8)
1801tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:8071s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:729s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:45:55 (UTC+8)
1800tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:7890s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:548s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:42:54 (UTC+8)
1799tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:7709s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:366s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-11:39:53 (UTC+8)
1798tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:7528s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:185s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:36:51 (UTC+8)
1797tasks done=3116/12141 running=2 failed=0 skipped=0 pending=9023 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:7347s;cuda2:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s42:4s | GPU cuda1:7751/24564MiB:54%;cuda2:3985/24564MiB:100%260818-11:33:50 (UTC+8)
1796tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:7166s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:4446s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:30:49 (UTC+8)
1795tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:6985s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:4266s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:27:48 (UTC+8)
1794tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:6803s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:4084s | GPU cuda1:7751/24564MiB:50%;cuda2:11435/24564MiB:100%260818-11:24:47 (UTC+8)
1793tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:6622s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:3903s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-11:21:45 (UTC+8)
1792tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:6440s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:3721s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:18:44 (UTC+8)
1791tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:6259s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:3540s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:15:43 (UTC+8)
1790tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:6078s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:3359s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-11:12:41 (UTC+8)
1789tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:5897s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:3178s | GPU cuda1:7751/24564MiB:50%;cuda2:11435/24564MiB:100%260818-11:09:40 (UTC+8)
1788tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:5716s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:2997s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-11:06:39 (UTC+8)
1787tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:5535s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:2816s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-11:03:38 (UTC+8)
1786tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:5353s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:2634s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-11:00:37 (UTC+8)
1785tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:5172s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:2453s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-10:57:36 (UTC+8)
1784tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:4991s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:2272s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-10:54:34 (UTC+8)
1783tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:4810s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:2091s | GPU cuda1:7751/24564MiB:51%;cuda2:11435/24564MiB:100%260818-10:51:33 (UTC+8)
1782tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:4628s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:1909s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-10:48:32 (UTC+8)
1781tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:4446s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:1727s | GPU cuda1:7751/24564MiB:89%;cuda2:11435/24564MiB:100%260818-10:45:30 (UTC+8)
1780tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:4265s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:1546s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-10:42:29 (UTC+8)
1779tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:4084s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:1365s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-10:39:27 (UTC+8)
1778tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:3903s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:1184s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-10:36:26 (UTC+8)
1777tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:3720s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:1001s | GPU cuda1:7751/24564MiB:78%;cuda2:11435/24564MiB:100%260818-10:33:24 (UTC+8)
1776tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:3539s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:820s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-10:30:23 (UTC+8)
1775tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:3358s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:639s | GPU cuda1:7751/24564MiB:53%;cuda2:11435/24564MiB:100%260818-10:27:22 (UTC+8)
1774tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:3177s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:458s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260818-10:24:21 (UTC+8)
1773tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:2995s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:276s | GPU cuda1:7751/24564MiB:54%;cuda2:11435/24564MiB:100%260818-10:21:19 (UTC+8)
1772tasks done=3115/12141 running=2 failed=0 skipped=0 pending=9024 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:2813s;cuda2:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s42:94s | GPU cuda1:7751/24564MiB:48%;cuda2:11435/24564MiB:99%260818-10:18:17 (UTC+8)
1771tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:2632s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:4560s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260818-10:15:15 (UTC+8)
1770tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:2450s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:4379s | GPU cuda1:7751/24564MiB:45%;cuda2:11435/24564MiB:100%260818-10:12:14 (UTC+8)
1769tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:2269s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:4197s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260818-10:09:13 (UTC+8)
1768tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:2088s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:4016s | GPU cuda1:7751/24564MiB:48%;cuda2:11435/24564MiB:100%260818-10:06:11 (UTC+8)
1767tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:1907s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:3835s | GPU cuda1:7751/24564MiB:64%;cuda2:11435/24564MiB:100%260818-10:03:10 (UTC+8)
1766tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:1725s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:3654s | GPU cuda1:7751/24564MiB:49%;cuda2:11435/24564MiB:100%260818-10:00:09 (UTC+8)
1765tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:1544s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:3473s | GPU cuda1:7751/24564MiB:48%;cuda2:11435/24564MiB:100%260818-09:57:08 (UTC+8)
1764tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:1363s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:3291s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260818-09:54:07 (UTC+8)
1763tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:1182s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:3110s | GPU cuda1:7751/24564MiB:52%;cuda2:11435/24564MiB:100%260818-09:51:05 (UTC+8)
1762tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:999s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:2927s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260818-09:48:04 (UTC+8)
1761tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:817s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:2746s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260818-09:45:01 (UTC+8)
1760tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:636s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:2565s | GPU cuda1:7751/24564MiB:42%;cuda2:11435/24564MiB:100%260818-09:42:00 (UTC+8)
1759tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:455s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:2383s | GPU cuda1:7751/24564MiB:48%;cuda2:11435/24564MiB:100%260818-09:38:59 (UTC+8)
1758tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:274s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:2202s | GPU cuda1:7751/24564MiB:47%;cuda2:11435/24564MiB:100%260818-09:35:57 (UTC+8)
1757tasks done=3113/12141 running=2 failed=0 skipped=0 pending=9026 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:duo_bank:duo_bank__llama3_8b__wikics__s666:92s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:2021s | GPU cuda1:7751/24564MiB:49%;cuda2:11435/24564MiB:100%260818-09:32:56 (UTC+8)
1756tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:2314s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:1840s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-09:29:55 (UTC+8)
1755tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:2132s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:1658s | GPU cuda1:7851/24564MiB:93%;cuda2:11435/24564MiB:77%260818-09:26:54 (UTC+8)
1754tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:1951s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:1477s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-09:23:52 (UTC+8)
1753tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:1769s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:1296s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-09:20:51 (UTC+8)
1752tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:1588s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:1114s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-09:17:49 (UTC+8)
1751tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:1407s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:933s | GPU cuda1:7851/24564MiB:90%;cuda2:11435/24564MiB:100%260818-09:14:48 (UTC+8)
1750tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:1224s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:750s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:98%260818-09:11:47 (UTC+8)
1749tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:1043s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:569s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-09:08:44 (UTC+8)
1748tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:862s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:388s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-09:05:43 (UTC+8)
1747tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:681s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:207s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:99%260818-09:02:42 (UTC+8)
1746tasks done=3111/12141 running=2 failed=0 skipped=0 pending=9028 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:499s;cuda2:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s42:25s | GPU cuda1:7851/24564MiB:100%;cuda2:11303/24564MiB:46%260818-08:59:41 (UTC+8)
1745tasks done=3110/12141 running=2 failed=0 skipped=0 pending=9029 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:318s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:3974s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-08:56:39 (UTC+8)
1744tasks done=3110/12141 running=2 failed=0 skipped=0 pending=9029 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s666:137s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:3793s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:53:38 (UTC+8)
1743tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:6462s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:3612s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:50:37 (UTC+8)
1742tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:6281s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:3431s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:47:36 (UTC+8)
1741tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:6099s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:3250s | GPU cuda1:11839/24564MiB:95%;cuda2:11435/24564MiB:100%260818-08:44:35 (UTC+8)
1740tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:5918s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:3069s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:41:33 (UTC+8)
1739tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:5737s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:2887s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:38:32 (UTC+8)
1738tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:5556s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:2706s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:35:31 (UTC+8)
1737tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:5375s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:2525s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:32:30 (UTC+8)
1736tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:5193s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:2344s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:29:29 (UTC+8)
1735tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:5012s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:2163s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:26:27 (UTC+8)
1734tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:4831s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:1982s | GPU cuda1:11839/24564MiB:99%;cuda2:11435/24564MiB:100%260818-08:23:26 (UTC+8)
1733tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:4650s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:1801s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:20:25 (UTC+8)
1732tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:4469s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:1619s | GPU cuda1:11839/24564MiB:99%;cuda2:11435/24564MiB:100%260818-08:17:24 (UTC+8)
1731tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:4288s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:1438s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:14:23 (UTC+8)
1730tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:4106s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:1256s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:11:22 (UTC+8)
1729tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:3925s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:1076s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:08:20 (UTC+8)
1728tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:3744s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:894s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-08:05:19 (UTC+8)
1727tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:3563s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:713s | GPU cuda1:11839/24564MiB:99%;cuda2:11435/24564MiB:100%260818-08:02:18 (UTC+8)
1726tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:3381s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:532s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:59:17 (UTC+8)
1725tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:3200s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:351s | GPU cuda1:11839/24564MiB:99%;cuda2:11435/24564MiB:100%260818-07:56:15 (UTC+8)
1724tasks done=3109/12141 running=2 failed=0 skipped=0 pending=9030 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:3019s;cuda2:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s42:169s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:53:14 (UTC+8)
1723tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:2838s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:5274s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:50:13 (UTC+8)
1722tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:2657s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:5093s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:47:12 (UTC+8)
1721tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:2476s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:4911s | GPU cuda1:11839/24564MiB:99%;cuda2:11435/24564MiB:100%260818-07:44:11 (UTC+8)
1720tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:2294s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:4730s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:41:10 (UTC+8)
1719tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:2113s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:4549s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:38:08 (UTC+8)
1718tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:1932s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:4367s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:35:07 (UTC+8)
1717tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:1751s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:4186s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:32:06 (UTC+8)
1716tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:1570s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:4006s | GPU cuda1:11839/24564MiB:98%;cuda2:11435/24564MiB:100%260818-07:29:05 (UTC+8)
1715tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:1389s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:3825s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:26:04 (UTC+8)
1714tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:1208s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:3644s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:23:03 (UTC+8)
1713tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:1026s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:3462s | GPU cuda1:11839/24564MiB:95%;cuda2:11435/24564MiB:100%260818-07:20:02 (UTC+8)
1712tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:845s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:3281s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:17:00 (UTC+8)
1711tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:664s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:3100s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:13:59 (UTC+8)
1710tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:483s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:2919s | GPU cuda1:11839/24564MiB:100%;cuda2:11435/24564MiB:100%260818-07:10:59 (UTC+8)
1709tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:301s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:2737s | GPU cuda1:11839/24564MiB:99%;cuda2:11435/24564MiB:100%260818-07:07:57 (UTC+8)
1708tasks done=3107/12141 running=2 failed=0 skipped=0 pending=9032 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s666:120s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:2556s | GPU cuda1:7663/24564MiB:25%;cuda2:11435/24564MiB:100%260818-07:04:56 (UTC+8)
1707tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:2097s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:2375s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-07:01:55 (UTC+8)
1706tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:1915s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:2194s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-06:58:54 (UTC+8)
1705tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:1734s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:2012s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:55:52 (UTC+8)
1704tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:1553s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:1831s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-06:52:51 (UTC+8)
1703tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:1372s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:1650s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:49:49 (UTC+8)
1702tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:1191s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:1469s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:46:49 (UTC+8)
1701tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:1009s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:1288s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:43:48 (UTC+8)
1700tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:828s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:1106s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-06:40:46 (UTC+8)
1699tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:647s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:925s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-06:37:45 (UTC+8)
1698tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:466s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:744s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:34:44 (UTC+8)
1697tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:285s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:564s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-06:31:43 (UTC+8)
1696tasks done=3105/12141 running=2 failed=0 skipped=0 pending=9034 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-b__llama3_8b__wikics__s666:104s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:383s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:28:42 (UTC+8)
1695tasks done=3104/12141 running=2 failed=0 skipped=0 pending=9035 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:2079s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:201s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-06:25:41 (UTC+8)
1694tasks done=3104/12141 running=2 failed=0 skipped=0 pending=9035 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:1898s;cuda2:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s42:20s | GPU cuda1:7851/24564MiB:95%;cuda2:11303/24564MiB:100%260818-06:22:40 (UTC+8)
1693tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:1717s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:5100s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:19:39 (UTC+8)
1692tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:1535s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:4919s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:16:37 (UTC+8)
1691tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:1354s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:4738s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-06:13:36 (UTC+8)
1690tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:1173s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:4556s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:10:35 (UTC+8)
1689tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:991s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:4375s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:07:34 (UTC+8)
1688tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:810s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:4193s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:04:32 (UTC+8)
1687tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:629s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:4012s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-06:01:31 (UTC+8)
1686tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:447s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:3831s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-05:58:30 (UTC+8)
1685tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:266s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:3650s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-05:55:28 (UTC+8)
1684tasks done=3103/12141 running=2 failed=0 skipped=0 pending=9036 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-1024-a__llama3_8b__wikics__s666:85s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:3469s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-05:52:27 (UTC+8)
1683tasks done=3101/12141 running=2 failed=0 skipped=0 pending=9038 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s666:1247s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:3288s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:49:26 (UTC+8)
1682tasks done=3101/12141 running=2 failed=0 skipped=0 pending=9038 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s666:1066s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:3107s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:46:25 (UTC+8)
1681tasks done=3101/12141 running=2 failed=0 skipped=0 pending=9038 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s666:885s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:2926s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:43:24 (UTC+8)
1680tasks done=3101/12141 running=2 failed=0 skipped=0 pending=9038 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s666:704s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:2745s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:40:23 (UTC+8)
1679tasks done=3101/12141 running=2 failed=0 skipped=0 pending=9038 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s666:523s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:2564s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:37:22 (UTC+8)
1678tasks done=3101/12141 running=2 failed=0 skipped=0 pending=9038 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s666:342s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:2383s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:34:21 (UTC+8)
1677tasks done=3101/12141 running=2 failed=0 skipped=0 pending=9038 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-b__llama3_8b__wikics__s666:160s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:2201s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:31:20 (UTC+8)
1676tasks done=3100/12141 running=2 failed=0 skipped=0 pending=9039 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s666:1320s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:2020s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:28:18 (UTC+8)
1675tasks done=3100/12141 running=2 failed=0 skipped=0 pending=9039 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s666:1138s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:1838s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:25:17 (UTC+8)
1674tasks done=3100/12141 running=2 failed=0 skipped=0 pending=9039 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s666:957s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:1657s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:22:16 (UTC+8)
1673tasks done=3100/12141 running=2 failed=0 skipped=0 pending=9039 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s666:776s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:1476s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:19:15 (UTC+8)
1672tasks done=3100/12141 running=2 failed=0 skipped=0 pending=9039 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s666:595s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:1295s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:16:14 (UTC+8)
1671tasks done=3100/12141 running=2 failed=0 skipped=0 pending=9039 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s666:414s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:1114s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:13:13 (UTC+8)
1670tasks done=3100/12141 running=2 failed=0 skipped=0 pending=9039 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s666:233s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:933s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:10:12 (UTC+8)
1669tasks done=3100/12141 running=2 failed=0 skipped=0 pending=9039 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-512-a__llama3_8b__wikics__s666:52s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:752s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:07:11 (UTC+8)
1668tasks done=3098/12141 running=2 failed=0 skipped=0 pending=9041 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s666:794s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:571s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-05:04:10 (UTC+8)
1667tasks done=3098/12141 running=2 failed=0 skipped=0 pending=9041 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s666:613s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:390s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-05:01:08 (UTC+8)
1666tasks done=3098/12141 running=2 failed=0 skipped=0 pending=9041 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s666:432s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:209s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260818-04:58:07 (UTC+8)
1665tasks done=3098/12141 running=2 failed=0 skipped=0 pending=9041 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s666:251s;cuda2:trace_context:trace_bank__Semantic-1024-a__llama3_8b__wikics__s42:28s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-04:55:06 (UTC+8)
1664tasks done=3096/12141 running=2 failed=0 skipped=0 pending=9043 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-b__llama3_8b__wikics__s666:70s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:3185s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-04:52:05 (UTC+8)
1663tasks done=3095/12141 running=2 failed=0 skipped=0 pending=9044 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s666:809s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:3003s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-04:49:04 (UTC+8)
1662tasks done=3095/12141 running=2 failed=0 skipped=0 pending=9044 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s666:628s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:2823s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260818-04:46:03 (UTC+8)
1661tasks done=3095/12141 running=2 failed=0 skipped=0 pending=9044 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s666:447s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:2641s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260818-04:43:02 (UTC+8)
1660tasks done=3095/12141 running=2 failed=0 skipped=0 pending=9044 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s666:266s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:2460s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260818-04:40:01 (UTC+8)
1659tasks done=3095/12141 running=2 failed=0 skipped=0 pending=9044 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-256-a__llama3_8b__wikics__s666:85s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:2279s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-04:37:00 (UTC+8)
1658tasks done=3093/12141 running=2 failed=0 skipped=0 pending=9046 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s666:700s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:2097s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-04:33:58 (UTC+8)
1657tasks done=3093/12141 running=2 failed=0 skipped=0 pending=9046 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s666:519s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:1916s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260818-04:30:56 (UTC+8)
1656tasks done=3093/12141 running=2 failed=0 skipped=0 pending=9046 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s666:338s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:1735s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260818-04:27:55 (UTC+8)
1655tasks done=3093/12141 running=2 failed=0 skipped=0 pending=9046 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-b__llama3_8b__wikics__s666:157s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:1554s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-04:24:54 (UTC+8)
1654tasks done=3092/12141 running=2 failed=0 skipped=0 pending=9047 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s666:770s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:1372s | GPU cuda1:7851/24564MiB:89%;cuda2:11435/24564MiB:100%260818-04:21:53 (UTC+8)
1653tasks done=3092/12141 running=2 failed=0 skipped=0 pending=9047 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s666:589s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:1191s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260818-04:18:52 (UTC+8)
1652tasks done=3092/12141 running=2 failed=0 skipped=0 pending=9047 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s666:408s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:1010s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-04:15:51 (UTC+8)
1651tasks done=3092/12141 running=2 failed=0 skipped=0 pending=9047 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s666:227s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:829s | GPU cuda1:7851/24564MiB:98%;cuda2:11435/24564MiB:100%260818-04:12:50 (UTC+8)
1650tasks done=3092/12141 running=2 failed=0 skipped=0 pending=9047 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__HopQuota-128-a__llama3_8b__wikics__s666:46s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:648s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-04:09:49 (UTC+8)
1649tasks done=3090/12141 running=2 failed=0 skipped=0 pending=9049 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:4354s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:467s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-04:06:48 (UTC+8)
1648tasks done=3090/12141 running=2 failed=0 skipped=0 pending=9049 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:4173s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:286s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-04:03:47 (UTC+8)
1647tasks done=3090/12141 running=2 failed=0 skipped=0 pending=9049 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:3991s;cuda2:trace_context:trace_bank__Semantic-512-b__llama3_8b__wikics__s42:105s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-04:00:45 (UTC+8)
1646tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:3810s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:3260s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-03:57:44 (UTC+8)
1645tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:3629s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:3079s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-03:54:43 (UTC+8)
1644tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:3448s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:2897s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-03:51:42 (UTC+8)
1643tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:3267s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:2716s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-03:48:41 (UTC+8)
1642tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:3085s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:2535s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-03:45:39 (UTC+8)
1641tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:2904s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:2354s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-03:42:38 (UTC+8)
1640tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:2723s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:2172s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-03:39:37 (UTC+8)
1639tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:2542s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:1991s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-03:36:36 (UTC+8)
1638tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:2361s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:1810s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-03:33:34 (UTC+8)
1637tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:2180s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:1629s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-03:30:33 (UTC+8)
1636tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:1999s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:1448s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-03:27:32 (UTC+8)
1635tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:1818s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:1267s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-03:24:31 (UTC+8)
1634tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:1636s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:1085s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-03:21:30 (UTC+8)
1633tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:1455s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:904s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-03:18:28 (UTC+8)
1632tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:1273s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:723s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-03:15:27 (UTC+8)
1631tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:1092s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:541s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-03:12:26 (UTC+8)
1630tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:911s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:360s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-03:09:24 (UTC+8)
1629tasks done=3089/12141 running=2 failed=0 skipped=0 pending=9050 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:729s;cuda2:trace_context:trace_bank__Semantic-512-a__llama3_8b__wikics__s42:178s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-03:06:23 (UTC+8)
1628tasks done=3087/12141 running=2 failed=0 skipped=0 pending=9052 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:548s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:2009s | GPU cuda1:7851/24564MiB:96%;cuda2:3597/24564MiB:100%260818-03:03:22 (UTC+8)
1627tasks done=3087/12141 running=2 failed=0 skipped=0 pending=9052 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:367s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:1828s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-03:00:21 (UTC+8)
1626tasks done=3087/12141 running=2 failed=0 skipped=0 pending=9052 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:186s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:1647s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-02:57:20 (UTC+8)
1625tasks done=3087/12141 running=2 failed=0 skipped=0 pending=9052 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-b__llama3_8b__wikics__s666:5s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:1466s | GPU cuda1:401/24564MiB:0%;cuda2:11435/24564MiB:100%260818-02:54:19 (UTC+8)
1624tasks done=3086/12141 running=2 failed=0 skipped=0 pending=9053 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:4332s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:1285s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-02:51:18 (UTC+8)
1623tasks done=3086/12141 running=2 failed=0 skipped=0 pending=9053 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:4151s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:1104s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-02:48:16 (UTC+8)
1622tasks done=3086/12141 running=2 failed=0 skipped=0 pending=9053 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:3970s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:923s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-02:45:15 (UTC+8)
1621tasks done=3086/12141 running=2 failed=0 skipped=0 pending=9053 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:3788s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:741s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-02:42:14 (UTC+8)
1620tasks done=3086/12141 running=2 failed=0 skipped=0 pending=9053 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:3607s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:560s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-02:39:13 (UTC+8)
1619tasks done=3086/12141 running=2 failed=0 skipped=0 pending=9053 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:3425s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:379s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-02:36:11 (UTC+8)
1618tasks done=3086/12141 running=2 failed=0 skipped=0 pending=9053 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:3244s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:197s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-02:33:10 (UTC+8)
1617tasks done=3086/12141 running=2 failed=0 skipped=0 pending=9053 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:3063s;cuda2:trace_context:trace_bank__Semantic-128-b__llama3_8b__wikics__s42:16s | GPU cuda1:7851/24564MiB:0%;cuda2:11279/24564MiB:100%260818-02:30:09 (UTC+8)
1616tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:2882s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:1844s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-02:27:08 (UTC+8)
1615tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:2701s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:1663s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-02:24:07 (UTC+8)
1614tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:2519s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:1482s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-02:21:06 (UTC+8)
1613tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:2338s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:1300s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-02:18:04 (UTC+8)
1612tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:2157s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:1119s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-02:15:02 (UTC+8)
1611tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:1976s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:938s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260818-02:12:01 (UTC+8)
1610tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:1795s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:757s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-02:09:00 (UTC+8)
1609tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:1614s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:576s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-02:05:59 (UTC+8)
1608tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:1432s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:394s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-02:02:58 (UTC+8)
1607tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:1251s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:213s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:59:57 (UTC+8)
1606tasks done=3085/12141 running=2 failed=0 skipped=0 pending=9054 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:1069s;cuda2:trace_context:trace_bank__Semantic-128-a__llama3_8b__wikics__s42:32s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:56:55 (UTC+8)
1605tasks done=3083/12141 running=2 failed=0 skipped=0 pending=9056 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:888s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:5108s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-01:53:54 (UTC+8)
1604tasks done=3083/12141 running=2 failed=0 skipped=0 pending=9056 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:707s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:4927s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-01:50:53 (UTC+8)
1603tasks done=3083/12141 running=2 failed=0 skipped=0 pending=9056 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:526s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:4746s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:47:51 (UTC+8)
1602tasks done=3083/12141 running=2 failed=0 skipped=0 pending=9056 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:345s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:4564s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-01:44:50 (UTC+8)
1601tasks done=3083/12141 running=2 failed=0 skipped=0 pending=9056 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-1024-a__llama3_8b__wikics__s666:164s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:4383s | GPU cuda1:7851/24564MiB:96%;cuda2:11435/24564MiB:100%260818-01:41:49 (UTC+8)
1600tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:3671s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:4202s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:38:48 (UTC+8)
1599tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:3489s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:4021s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:35:47 (UTC+8)
1598tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:3308s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:3840s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:32:46 (UTC+8)
1597tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:3127s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:3659s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:29:45 (UTC+8)
1596tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:2944s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:3475s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:26:44 (UTC+8)
1595tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:2763s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:3294s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:23:40 (UTC+8)
1594tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:2581s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:3113s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:20:39 (UTC+8)
1593tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:2400s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:2932s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:17:37 (UTC+8)
1592tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:2219s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:2751s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-01:14:36 (UTC+8)
1591tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:2038s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:2570s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:11:35 (UTC+8)
1590tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:1857s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:2389s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-01:08:34 (UTC+8)
1589tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:1676s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:2208s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-01:05:33 (UTC+8)
1588tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:1495s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:2027s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-01:02:32 (UTC+8)
1587tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:1314s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:1846s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:59:31 (UTC+8)
1586tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:1133s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:1665s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-00:56:30 (UTC+8)
1585tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:952s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:1483s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:53:29 (UTC+8)
1584tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:770s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:1302s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:50:28 (UTC+8)
1583tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:589s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:1121s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:47:27 (UTC+8)
1582tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:408s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:940s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:44:26 (UTC+8)
1581tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:227s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:759s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-00:41:25 (UTC+8)
1580tasks done=3081/12141 running=2 failed=0 skipped=0 pending=9058 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-b__llama3_8b__wikics__s666:46s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:578s | GPU cuda1:7835/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:38:24 (UTC+8)
1579tasks done=3080/12141 running=2 failed=0 skipped=0 pending=9059 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:3536s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:397s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:35:23 (UTC+8)
1578tasks done=3080/12141 running=2 failed=0 skipped=0 pending=9059 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:3355s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:215s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:32:21 (UTC+8)
1577tasks done=3080/12141 running=2 failed=0 skipped=0 pending=9059 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:3174s;cuda2:trace_context:trace_bank__Random-1024-b__llama3_8b__wikics__s42:34s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:29:20 (UTC+8)
1576tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:2993s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:5108s | GPU cuda1:7851/24564MiB:100%;cuda2:11435/24564MiB:100%260818-00:26:19 (UTC+8)
1575tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:2811s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:4927s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:23:18 (UTC+8)
1574tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:2630s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:4746s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-00:20:17 (UTC+8)
1573tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:2449s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:4565s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:17:16 (UTC+8)
1572tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:2268s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:4384s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:14:15 (UTC+8)
1571tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:2087s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:4202s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-00:11:14 (UTC+8)
1570tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:1905s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:4021s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:08:12 (UTC+8)
1569tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:1724s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:3840s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260818-00:05:11 (UTC+8)
1568tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:1544s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:3659s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260818-00:02:10 (UTC+8)
1567tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:1362s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:3478s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:59:09 (UTC+8)
1566tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:1180s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:3295s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260817-23:56:08 (UTC+8)
1565tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:999s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:3114s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:53:05 (UTC+8)
1564tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:818s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:2933s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260817-23:50:04 (UTC+8)
1563tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:637s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:2752s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:47:03 (UTC+8)
1562tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:456s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:2571s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260817-23:44:02 (UTC+8)
1561tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:275s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:2390s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:41:01 (UTC+8)
1560tasks done=3079/12141 running=2 failed=0 skipped=0 pending=9060 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-512-a__llama3_8b__wikics__s666:93s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:2209s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260817-23:38:00 (UTC+8)
1559tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:3262s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:2028s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:34:59 (UTC+8)
1558tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:3081s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:1847s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:31:58 (UTC+8)
1557tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:2900s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:1666s | GPU cuda1:7851/24564MiB:46%;cuda2:11435/24564MiB:100%260817-23:28:57 (UTC+8)
1556tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:2719s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:1485s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260817-23:25:56 (UTC+8)
1555tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:2538s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:1303s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:22:55 (UTC+8)
1554tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:2357s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:1122s | GPU cuda1:7851/24564MiB:97%;cuda2:11435/24564MiB:100%260817-23:19:53 (UTC+8)
1553tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:2173s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:939s | GPU cuda1:7851/24564MiB:95%;cuda2:11435/24564MiB:100%260817-23:16:52 (UTC+8)
1552tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:1992s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:758s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:13:49 (UTC+8)
1551tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:1812s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:577s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:10:48 (UTC+8)
1550tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:1631s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:396s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:07:47 (UTC+8)
1549tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:1449s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:215s | GPU cuda1:7851/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:04:46 (UTC+8)
1548tasks done=3077/12141 running=2 failed=0 skipped=0 pending=9062 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:1268s;cuda2:trace_context:trace_bank__Random-1024-a__llama3_8b__wikics__s42:33s | GPU cuda1:8411/24564MiB:0%;cuda2:11435/24564MiB:100%260817-23:01:44 (UTC+8)
1547tasks done=3075/12141 running=2 failed=0 skipped=0 pending=9064 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:1087s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:3157s | GPU cuda1:8491/24564MiB:15%;cuda2:11435/24564MiB:100%260817-22:58:43 (UTC+8)
1546tasks done=3075/12141 running=2 failed=0 skipped=0 pending=9064 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:906s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:2976s | GPU cuda1:8489/24564MiB:15%;cuda2:11435/24564MiB:100%260817-22:55:42 (UTC+8)
1545tasks done=3075/12141 running=2 failed=0 skipped=0 pending=9064 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:725s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:2795s | GPU cuda1:8489/24564MiB:15%;cuda2:11435/24564MiB:100%260817-22:52:41 (UTC+8)
1544tasks done=3075/12141 running=2 failed=0 skipped=0 pending=9064 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:543s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:2614s | GPU cuda1:8489/24564MiB:15%;cuda2:11435/24564MiB:100%260817-22:49:40 (UTC+8)
1543tasks done=3075/12141 running=2 failed=0 skipped=0 pending=9064 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:362s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:2433s | GPU cuda1:8489/24564MiB:99%;cuda2:11435/24564MiB:100%260817-22:46:39 (UTC+8)
1542tasks done=3075/12141 running=2 failed=0 skipped=0 pending=9064 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-b__llama3_8b__wikics__s666:178s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:2249s | GPU cuda1:8489/24564MiB:98%;cuda2:11435/24564MiB:100%260817-22:43:38 (UTC+8)
1541tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:3352s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:2067s | GPU cuda1:8489/24564MiB:98%;cuda2:11435/24564MiB:100%260817-22:40:34 (UTC+8)
1540tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:3171s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:1886s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-22:37:32 (UTC+8)
1539tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:2990s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:1705s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-22:34:31 (UTC+8)
1538tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:2809s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:1524s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-22:31:30 (UTC+8)
1537tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:2628s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:1343s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-22:28:29 (UTC+8)
1536tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:2446s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:1161s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-22:25:28 (UTC+8)
1535tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:2265s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:980s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-22:22:26 (UTC+8)
1534tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:2084s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:799s | GPU cuda1:8489/24564MiB:98%;cuda2:11435/24564MiB:100%260817-22:19:25 (UTC+8)
1533tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:1903s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:618s | GPU cuda1:8413/24564MiB:97%;cuda2:11435/24564MiB:100%260817-22:16:24 (UTC+8)
1532tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:1722s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:437s | GPU cuda1:8489/24564MiB:98%;cuda2:11435/24564MiB:100%260817-22:13:23 (UTC+8)
1531tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:1536s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:251s | GPU cuda1:8489/24564MiB:15%;cuda2:11435/24564MiB:100%260817-22:10:22 (UTC+8)
1530tasks done=3074/12141 running=2 failed=0 skipped=0 pending=9065 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:1355s;cuda2:trace_context:trace_bank__Random-512-b__llama3_8b__wikics__s42:70s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-22:07:16 (UTC+8)
1529tasks done=3073/12141 running=2 failed=0 skipped=0 pending=9066 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:1174s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:3192s | GPU cuda1:8489/24564MiB:15%;cuda2:11435/24564MiB:100%260817-22:04:15 (UTC+8)
1528tasks done=3073/12141 running=2 failed=0 skipped=0 pending=9066 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:993s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:3011s | GPU cuda1:8489/24564MiB:15%;cuda2:11435/24564MiB:100%260817-22:01:14 (UTC+8)
1527tasks done=3073/12141 running=2 failed=0 skipped=0 pending=9066 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:811s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:2830s | GPU cuda1:8489/24564MiB:15%;cuda2:11435/24564MiB:100%260817-21:58:13 (UTC+8)
1526tasks done=3073/12141 running=2 failed=0 skipped=0 pending=9066 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:630s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:2648s | GPU cuda1:8489/24564MiB:15%;cuda2:11435/24564MiB:100%260817-21:55:11 (UTC+8)
1525tasks done=3073/12141 running=2 failed=0 skipped=0 pending=9066 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:448s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:2467s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-21:52:10 (UTC+8)
1524tasks done=3073/12141 running=2 failed=0 skipped=0 pending=9066 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:267s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:2286s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-21:49:08 (UTC+8)
1523tasks done=3073/12141 running=2 failed=0 skipped=0 pending=9066 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-256-a__llama3_8b__wikics__s666:86s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:2104s | GPU cuda1:8489/24564MiB:98%;cuda2:11435/24564MiB:100%260817-21:46:07 (UTC+8)
1522tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:3074s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:1923s | GPU cuda1:8489/24564MiB:99%;cuda2:11435/24564MiB:100%260817-21:43:06 (UTC+8)
1521tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:2892s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:1742s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-21:40:05 (UTC+8)
1520tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:2712s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:1561s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-21:37:04 (UTC+8)
1519tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:2530s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:1380s | GPU cuda1:8489/24564MiB:13%;cuda2:11435/24564MiB:100%260817-21:34:03 (UTC+8)
1518tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:2349s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:1199s | GPU cuda1:8489/24564MiB:99%;cuda2:11435/24564MiB:100%260817-21:31:01 (UTC+8)
1517tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:2168s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:1018s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-21:28:00 (UTC+8)
1516tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:1987s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:837s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-21:24:59 (UTC+8)
1515tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:1806s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:656s | GPU cuda1:8489/24564MiB:99%;cuda2:11435/24564MiB:100%260817-21:21:58 (UTC+8)
1514tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:1625s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:475s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-21:18:57 (UTC+8)
1513tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:1442s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:292s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-21:15:56 (UTC+8)
1512tasks done=3071/12141 running=2 failed=0 skipped=0 pending=9068 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:1261s;cuda2:trace_context:trace_bank__Random-512-a__llama3_8b__wikics__s42:111s | GPU cuda1:8489/24564MiB:99%;cuda2:11435/24564MiB:100%260817-21:12:54 (UTC+8)
1511tasks done=3069/12141 running=2 failed=0 skipped=0 pending=9070 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:1080s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:2310s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-21:09:53 (UTC+8)
1510tasks done=3069/12141 running=2 failed=0 skipped=0 pending=9070 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:899s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:2129s | GPU cuda1:8489/24564MiB:13%;cuda2:11435/24564MiB:100%260817-21:06:52 (UTC+8)
1509tasks done=3069/12141 running=2 failed=0 skipped=0 pending=9070 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:717s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:1947s | GPU cuda1:8489/24564MiB:98%;cuda2:11435/24564MiB:100%260817-21:03:51 (UTC+8)
1508tasks done=3069/12141 running=2 failed=0 skipped=0 pending=9070 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:535s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:1765s | GPU cuda1:8489/24564MiB:99%;cuda2:11435/24564MiB:100%260817-21:00:48 (UTC+8)
1507tasks done=3069/12141 running=2 failed=0 skipped=0 pending=9070 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:354s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:1584s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-20:57:47 (UTC+8)
1506tasks done=3069/12141 running=2 failed=0 skipped=0 pending=9070 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-b__llama3_8b__wikics__s666:172s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:1402s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-20:54:46 (UTC+8)
1505tasks done=3068/12141 running=2 failed=0 skipped=0 pending=9071 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:3093s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:1221s | GPU cuda1:8489/24564MiB:14%;cuda2:11435/24564MiB:100%260817-20:51:43 (UTC+8)
1504tasks done=3068/12141 running=2 failed=0 skipped=0 pending=9071 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:2912s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:1039s | GPU cuda1:8487/24564MiB:97%;cuda2:11435/24564MiB:100%260817-20:48:42 (UTC+8)
1503tasks done=3068/12141 running=2 failed=0 skipped=0 pending=9071 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:2731s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:858s | GPU cuda1:8511/24564MiB:15%;cuda2:11435/24564MiB:100%260817-20:45:41 (UTC+8)
1502tasks done=3068/12141 running=2 failed=0 skipped=0 pending=9071 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:2550s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:677s | GPU cuda1:8511/24564MiB:14%;cuda2:11435/24564MiB:100%260817-20:42:40 (UTC+8)
1501tasks done=3068/12141 running=2 failed=0 skipped=0 pending=9071 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:2368s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:496s | GPU cuda1:8511/24564MiB:98%;cuda2:11435/24564MiB:100%260817-20:39:39 (UTC+8)
1500tasks done=3068/12141 running=2 failed=0 skipped=0 pending=9071 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:2187s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:314s | GPU cuda1:8511/24564MiB:15%;cuda2:11435/24564MiB:100%260817-20:36:37 (UTC+8)
1499tasks done=3068/12141 running=2 failed=0 skipped=0 pending=9071 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:2006s;cuda2:trace_context:trace_bank__Random-256-b__llama3_8b__wikics__s42:133s | GPU cuda1:8511/24564MiB:98%;cuda2:11435/24564MiB:100%260817-20:33:36 (UTC+8)
1498tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:1825s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:2330s | GPU cuda1:8511/24564MiB:14%;cuda2:11435/24564MiB:100%260817-20:30:35 (UTC+8)
1497tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:1643s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:2149s | GPU cuda1:8511/24564MiB:15%;cuda2:11435/24564MiB:100%260817-20:27:34 (UTC+8)
1496tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:1462s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:1968s | GPU cuda1:8511/24564MiB:14%;cuda2:11435/24564MiB:100%260817-20:24:33 (UTC+8)
1495tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:1281s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:1786s | GPU cuda1:8511/24564MiB:15%;cuda2:11435/24564MiB:100%260817-20:21:31 (UTC+8)
1494tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:1100s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:1605s | GPU cuda1:8511/24564MiB:98%;cuda2:11435/24564MiB:100%260817-20:18:30 (UTC+8)
1493tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:919s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:1424s | GPU cuda1:8511/24564MiB:14%;cuda2:11435/24564MiB:100%260817-20:15:29 (UTC+8)
1492tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:738s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:1243s | GPU cuda1:8511/24564MiB:99%;cuda2:11435/24564MiB:100%260817-20:12:28 (UTC+8)
1491tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:557s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:1062s | GPU cuda1:8511/24564MiB:14%;cuda2:11435/24564MiB:100%260817-20:09:27 (UTC+8)
1490tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:374s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:880s | GPU cuda1:8511/24564MiB:14%;cuda2:11435/24564MiB:100%260817-20:06:26 (UTC+8)
1489tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:193s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:699s | GPU cuda1:8511/24564MiB:98%;cuda2:11435/24564MiB:100%260817-20:03:23 (UTC+8)
1488tasks done=3067/12141 running=2 failed=0 skipped=0 pending=9072 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Diverse-128-a__llama3_8b__wikics__s666:12s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:518s | GPU cuda1:8325/24564MiB:30%;cuda2:11435/24564MiB:100%260817-20:00:22 (UTC+8)
1487tasks done=3065/12141 running=2 failed=0 skipped=0 pending=9074 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:2496s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:336s | GPU cuda1:8511/24564MiB:97%;cuda2:11435/24564MiB:100%260817-19:57:21 (UTC+8)
1486tasks done=3065/12141 running=2 failed=0 skipped=0 pending=9074 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:2315s;cuda2:trace_context:trace_bank__Random-256-a__llama3_8b__wikics__s42:155s | GPU cuda1:8511/24564MiB:100%;cuda2:11435/24564MiB:100%260817-19:54:20 (UTC+8)
1485tasks done=3063/12141 running=2 failed=0 skipped=0 pending=9076 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:2134s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s42:1957s | GPU cuda1:8511/24564MiB:97%;cuda2:11435/24564MiB:100%260817-19:51:19 (UTC+8)
1484tasks done=3063/12141 running=2 failed=0 skipped=0 pending=9076 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:1952s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s42:1776s | GPU cuda1:8511/24564MiB:97%;cuda2:11435/24564MiB:100%260817-19:48:18 (UTC+8)
1483tasks done=3063/12141 running=2 failed=0 skipped=0 pending=9076 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:1771s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s42:1595s | GPU cuda1:8511/24564MiB:97%;cuda2:11435/24564MiB:100%260817-19:45:17 (UTC+8)
1482tasks done=3063/12141 running=2 failed=0 skipped=0 pending=9076 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:1590s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s42:1414s | GPU cuda1:8511/24564MiB:100%;cuda2:11435/24564MiB:100%260817-19:42:16 (UTC+8)
1481tasks done=3063/12141 running=2 failed=0 skipped=0 pending=9076 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:1408s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s42:1231s | GPU cuda1:8511/24564MiB:98%;cuda2:11435/24564MiB:100%260817-19:39:15 (UTC+8)
1480tasks done=3063/12141 running=2 failed=0 skipped=0 pending=9076 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:1227s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s42:1050s | GPU cuda1:8511/24564MiB:100%;cuda2:11435/24564MiB:100%260817-19:36:12 (UTC+8)
1479tasks done=3063/12141 running=2 failed=0 skipped=0 pending=9076 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:1044s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s42:868s | GPU cuda1:8511/24564MiB:97%;cuda2:11435/24564MiB:100%260817-19:33:11 (UTC+8)
1478tasks done=3063/12141 running=2 failed=0 skipped=0 pending=9076 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:863s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s42:687s | GPU cuda1:8511/24564MiB:96%;cuda2:11435/24564MiB:100%260817-19:30:09 (UTC+8)
1477tasks done=3063/12141 running=2 failed=0 skipped=0 pending=9076 | results done=2000/8095 failed=0 skipped=0 | RUNNING cuda1:trace_context:trace_bank__Semantic-1024-b__llama3_8b__wikics__s666:682s;cuda2:trace_context:trace_bank__Random-128-b__llama3_8b__wikics__s42:506s | GPU cuda1:8511/24564MiB:100%;cuda2:11435/24564MiB:100%260817-19:27:08 (UTC+8)
#02

615-ACL-EX

▦ table
last update
260822-00:39:20
records
2000
type
text
latest
2026-08-22 00:39:19 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 20% cuda3:14/24564MiB 0%
ID数值接收时间
39872026-08-22 00:39:19 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 20% cuda3:14/24564MiB 0%260822-00:39:20 (UTC+8)
39862026-08-22 00:29:18 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 13% cuda3:14/24564MiB 0%260822-00:29:19 (UTC+8)
39852026-08-22 00:19:18 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 19% cuda3:14/24564MiB 0%260822-00:19:18 (UTC+8)
39842026-08-22 00:09:16 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 19% cuda3:14/24564MiB 0%260822-00:09:17 (UTC+8)
39832026-08-21 23:59:16 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 16% cuda3:14/24564MiB 0%260821-23:59:16 (UTC+8)
39822026-08-21 23:49:15 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 29% cuda3:14/24564MiB 0%260821-23:49:16 (UTC+8)
39812026-08-21 23:39:14 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 24% cuda3:14/24564MiB 0%260821-23:39:14 (UTC+8)
39802026-08-21 23:29:13 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 29% cuda3:14/24564MiB 0%260821-23:29:13 (UTC+8)
39792026-08-21 23:19:11 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 16% cuda3:14/24564MiB 0%260821-23:19:13 (UTC+8)
39782026-08-21 23:09:10 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 16% cuda3:14/24564MiB 0%260821-23:09:11 (UTC+8)
39772026-08-21 22:59:09 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 23% cuda3:14/24564MiB 0%260821-22:59:10 (UTC+8)
39762026-08-21 22:49:08 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 26% cuda3:14/24564MiB 0%260821-22:49:09 (UTC+8)
39752026-08-21 22:39:08 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 22% cuda3:14/24564MiB 0%260821-22:39:08 (UTC+8)
39742026-08-21 22:29:06 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 23% cuda3:14/24564MiB 0%260821-22:29:07 (UTC+8)
39732026-08-21 22:19:05 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 18% cuda3:14/24564MiB 0%260821-22:19:06 (UTC+8)
39722026-08-21 22:09:04 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 46% cuda3:14/24564MiB 0%260821-22:09:05 (UTC+8)
39712026-08-21 21:59:04 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 17% cuda3:14/24564MiB 0%260821-21:59:04 (UTC+8)
39702026-08-21 21:49:03 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 20% cuda3:14/24564MiB 0%260821-21:49:04 (UTC+8)
39692026-08-21 21:39:02 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 27% cuda3:14/24564MiB 0%260821-21:39:03 (UTC+8)
39682026-08-21 21:29:01 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 43% cuda3:14/24564MiB 0%260821-21:29:02 (UTC+8)
39672026-08-21 21:19:01 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 13% cuda3:14/24564MiB 0%260821-21:19:01 (UTC+8)
39662026-08-21 21:09:00 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 21% cuda3:14/24564MiB 0%260821-21:09:01 (UTC+8)
39652026-08-21 20:58:59 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 16% cuda3:14/24564MiB 0%260821-20:59:00 (UTC+8)
39642026-08-21 20:48:58 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 26% cuda3:14/24564MiB 0%260821-20:48:59 (UTC+8)
39632026-08-21 20:38:57 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 32% cuda3:14/24564MiB 0%260821-20:38:58 (UTC+8)
39622026-08-21 20:28:57 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 36% cuda3:14/24564MiB 0%260821-20:28:57 (UTC+8)
39612026-08-21 20:18:56 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 21% cuda3:14/24564MiB 0%260821-20:18:56 (UTC+8)
39602026-08-21 20:08:55 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 24% cuda3:14/24564MiB 0%260821-20:08:56 (UTC+8)
39592026-08-21 19:58:55 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 29% cuda3:14/24564MiB 0%260821-19:58:55 (UTC+8)
39582026-08-21 19:48:54 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 29% cuda3:14/24564MiB 0%260821-19:48:54 (UTC+8)
39572026-08-21 19:38:53 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 25% cuda3:14/24564MiB 0%260821-19:38:54 (UTC+8)
39562026-08-21 19:28:53 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 18% cuda3:14/24564MiB 0%260821-19:28:53 (UTC+8)
39552026-08-21 19:18:52 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 23% cuda3:14/24564MiB 0%260821-19:18:52 (UTC+8)
39542026-08-21 19:08:51 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 31% cuda3:14/24564MiB 0%260821-19:08:52 (UTC+8)
39532026-08-21 18:58:50 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 17% cuda3:14/24564MiB 0%260821-18:58:51 (UTC+8)
39522026-08-21 18:48:50 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 12% cuda3:14/24564MiB 0%260821-18:48:50 (UTC+8)
39512026-08-21 18:38:49 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 18% cuda3:14/24564MiB 0%260821-18:38:50 (UTC+8)
39502026-08-21 18:28:48 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 32% cuda3:14/24564MiB 0%260821-18:28:49 (UTC+8)
39492026-08-21 18:18:47 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 23% cuda3:14/24564MiB 0%260821-18:18:48 (UTC+8)
39482026-08-21 18:08:46 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 24% cuda3:14/24564MiB 0%260821-18:08:47 (UTC+8)
39472026-08-21 17:58:45 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 22% cuda3:14/24564MiB 0%260821-17:58:46 (UTC+8)
39462026-08-21 17:48:44 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 43% cuda3:14/24564MiB 0%260821-17:48:45 (UTC+8)
39452026-08-21 17:38:43 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 22% cuda3:14/24564MiB 0%260821-17:38:44 (UTC+8)
39442026-08-21 17:28:43 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 23% cuda3:14/24564MiB 0%260821-17:28:43 (UTC+8)
39432026-08-21 17:18:42 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 20% cuda3:14/24564MiB 0%260821-17:18:43 (UTC+8)
39422026-08-21 17:08:41 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 19% cuda3:14/24564MiB 0%260821-17:08:42 (UTC+8)
39412026-08-21 16:58:40 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 20% cuda3:14/24564MiB 0%260821-16:58:41 (UTC+8)
39402026-08-21 16:48:40 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 48% cuda3:14/24564MiB 0%260821-16:48:40 (UTC+8)
39392026-08-21 16:38:39 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 23% cuda3:14/24564MiB 0%260821-16:38:40 (UTC+8)
39382026-08-21 16:28:38 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 18% cuda3:14/24564MiB 0%260821-16:28:39 (UTC+8)
39372026-08-21 16:18:37 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 29% cuda3:14/24564MiB 0%260821-16:18:38 (UTC+8)
39362026-08-21 16:08:36 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 47% cuda3:14/24564MiB 0%260821-16:08:37 (UTC+8)
39352026-08-21 15:58:35 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 20% cuda3:14/24564MiB 0%260821-15:58:36 (UTC+8)
39342026-08-21 15:48:34 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 18% cuda3:14/24564MiB 0%260821-15:48:35 (UTC+8)
39332026-08-21 15:38:33 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 30% cuda3:14/24564MiB 0%260821-15:38:34 (UTC+8)
39322026-08-21 15:28:32 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 15% cuda3:14/24564MiB 0%260821-15:28:33 (UTC+8)
39312026-08-21 15:18:32 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 16% cuda3:14/24564MiB 0%260821-15:18:32 (UTC+8)
39302026-08-21 15:08:31 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 57% cuda3:14/24564MiB 0%260821-15:08:32 (UTC+8)
39292026-08-21 14:58:29 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 33% cuda3:14/24564MiB 0%260821-14:58:31 (UTC+8)
39282026-08-21 14:48:28 done=216/273 running=0 failed=4 pending=53 cuda1:7683/24564MiB 15% cuda3:14/24564MiB 0%260821-14:48:29 (UTC+8)
39272026-08-21 14:38:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-14:38:28 (UTC+8)
39262026-08-21 14:28:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-14:28:27 (UTC+8)
39252026-08-21 14:18:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-14:18:27 (UTC+8)
39242026-08-21 14:08:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 84% cuda3:14/24564MiB 0%260821-14:08:26 (UTC+8)
39232026-08-21 13:58:25 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260821-13:58:25 (UTC+8)
39222026-08-21 13:48:24 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260821-13:48:25 (UTC+8)
39212026-08-21 13:38:23 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260821-13:38:24 (UTC+8)
39202026-08-21 13:28:20 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 95% cuda3:14/24564MiB 0%260821-13:28:23 (UTC+8)
39192026-08-21 13:18:19 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260821-13:18:19 (UTC+8)
39182026-08-21 13:08:18 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 65% cuda3:14/24564MiB 0%260821-13:08:19 (UTC+8)
39172026-08-21 12:58:18 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260821-12:58:18 (UTC+8)
39162026-08-21 12:48:17 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260821-12:48:18 (UTC+8)
39152026-08-21 12:38:16 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260821-12:38:17 (UTC+8)
39142026-08-21 12:28:16 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260821-12:28:16 (UTC+8)
39132026-08-21 12:18:14 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 98% cuda3:14/24564MiB 0%260821-12:18:16 (UTC+8)
39122026-08-21 12:08:13 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:14/24564MiB 0%260821-12:08:14 (UTC+8)
39112026-08-21 11:58:12 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 63% cuda3:14/24564MiB 0%260821-11:58:13 (UTC+8)
39102026-08-21 11:48:12 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-11:48:12 (UTC+8)
39092026-08-21 11:38:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-11:38:12 (UTC+8)
39082026-08-21 11:28:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-11:28:11 (UTC+8)
39072026-08-21 11:18:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-11:18:10 (UTC+8)
39062026-08-21 11:08:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 49% cuda3:14/24564MiB 0%260821-11:08:09 (UTC+8)
39052026-08-21 10:58:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 91% cuda3:14/24564MiB 0%260821-10:58:08 (UTC+8)
39042026-08-21 10:48:07 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 88% cuda3:14/24564MiB 0%260821-10:48:08 (UTC+8)
39032026-08-21 10:38:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 74% cuda3:14/24564MiB 0%260821-10:38:07 (UTC+8)
39022026-08-21 10:28:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 79% cuda3:14/24564MiB 0%260821-10:28:06 (UTC+8)
39012026-08-21 10:18:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 81% cuda3:14/24564MiB 0%260821-10:18:05 (UTC+8)
39002026-08-21 10:08:03 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-10:08:04 (UTC+8)
38992026-08-21 09:58:03 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 89% cuda3:14/24564MiB 0%260821-09:58:03 (UTC+8)
38982026-08-21 09:48:02 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-09:48:02 (UTC+8)
38972026-08-21 09:38:01 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-09:38:02 (UTC+8)
38962026-08-21 09:28:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260821-09:28:01 (UTC+8)
38952026-08-21 09:17:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 91% cuda3:14/24564MiB 0%260821-09:18:00 (UTC+8)
38942026-08-21 09:07:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260821-09:07:58 (UTC+8)
38932026-08-21 08:57:57 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 87% cuda3:14/24564MiB 0%260821-08:57:58 (UTC+8)
38922026-08-21 08:47:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260821-08:47:57 (UTC+8)
38912026-08-21 08:37:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-08:37:56 (UTC+8)
38902026-08-21 08:27:55 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-08:27:55 (UTC+8)
38892026-08-21 08:17:54 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260821-08:17:55 (UTC+8)
38882026-08-21 08:07:53 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-08:07:54 (UTC+8)
38872026-08-21 07:57:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-07:57:53 (UTC+8)
38862026-08-21 07:47:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-07:47:52 (UTC+8)
38852026-08-21 07:37:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260821-07:37:51 (UTC+8)
38842026-08-21 07:27:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260821-07:27:51 (UTC+8)
38832026-08-21 07:17:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260821-07:17:50 (UTC+8)
38822026-08-21 07:07:48 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-07:07:49 (UTC+8)
38812026-08-21 06:57:48 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260821-06:57:48 (UTC+8)
38802026-08-21 06:47:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260821-06:47:48 (UTC+8)
38792026-08-21 06:37:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-06:37:47 (UTC+8)
38782026-08-21 06:27:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-06:27:47 (UTC+8)
38772026-08-21 06:17:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-06:17:46 (UTC+8)
38762026-08-21 06:07:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-06:07:45 (UTC+8)
38752026-08-21 05:57:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-05:57:44 (UTC+8)
38742026-08-21 05:47:43 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-05:47:44 (UTC+8)
38732026-08-21 05:37:43 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-05:37:43 (UTC+8)
38722026-08-21 05:27:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-05:27:43 (UTC+8)
38712026-08-21 05:17:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-05:17:42 (UTC+8)
38702026-08-21 05:07:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-05:07:42 (UTC+8)
38692026-08-21 04:57:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-04:57:41 (UTC+8)
38682026-08-21 04:47:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-04:47:40 (UTC+8)
38672026-08-21 04:37:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-04:37:40 (UTC+8)
38662026-08-21 04:27:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-04:27:39 (UTC+8)
38652026-08-21 04:17:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-04:17:39 (UTC+8)
38642026-08-21 04:07:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-04:07:38 (UTC+8)
38632026-08-21 03:57:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-03:57:37 (UTC+8)
38622026-08-21 03:47:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-03:47:36 (UTC+8)
38612026-08-21 03:37:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-03:37:36 (UTC+8)
38602026-08-21 03:27:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-03:27:35 (UTC+8)
38592026-08-21 03:17:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260821-03:17:35 (UTC+8)
38582026-08-21 03:07:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-03:07:34 (UTC+8)
38572026-08-21 02:57:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-02:57:33 (UTC+8)
38562026-08-21 02:47:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-02:47:32 (UTC+8)
38552026-08-21 02:37:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-02:37:32 (UTC+8)
38542026-08-21 02:27:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260821-02:27:31 (UTC+8)
38532026-08-21 02:17:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-02:17:30 (UTC+8)
38522026-08-21 02:07:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-02:07:30 (UTC+8)
38512026-08-21 01:57:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-01:57:29 (UTC+8)
38502026-08-21 01:47:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-01:47:28 (UTC+8)
38492026-08-21 01:37:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-01:37:27 (UTC+8)
38482026-08-21 01:27:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-01:27:27 (UTC+8)
38472026-08-21 01:17:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260821-01:17:26 (UTC+8)
38462026-08-21 01:07:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-01:07:25 (UTC+8)
38452026-08-21 00:57:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260821-00:57:24 (UTC+8)
38442026-08-21 00:47:23 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-00:47:24 (UTC+8)
38432026-08-21 00:37:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-00:37:23 (UTC+8)
38422026-08-21 00:27:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260821-00:27:22 (UTC+8)
38412026-08-21 00:17:20 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260821-00:17:22 (UTC+8)
38402026-08-21 00:07:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 94% cuda3:14/24564MiB 0%260821-00:07:20 (UTC+8)
38392026-08-20 23:57:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260820-23:57:19 (UTC+8)
38382026-08-20 23:47:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 4% cuda3:14/24564MiB 0%260820-23:47:18 (UTC+8)
38372026-08-20 23:37:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-23:37:17 (UTC+8)
38362026-08-20 23:27:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-23:27:16 (UTC+8)
38352026-08-20 23:17:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-23:17:14 (UTC+8)
38342026-08-20 23:07:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-23:07:14 (UTC+8)
38332026-08-20 22:57:12 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-22:57:13 (UTC+8)
38322026-08-20 22:47:12 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260820-22:47:12 (UTC+8)
38312026-08-20 22:37:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260820-22:37:11 (UTC+8)
38302026-08-20 22:27:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-22:27:08 (UTC+8)
38292026-08-20 22:17:07 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260820-22:17:07 (UTC+8)
38282026-08-20 22:07:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260820-22:07:07 (UTC+8)
38272026-08-20 21:57:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260820-21:57:06 (UTC+8)
38262026-08-20 21:47:01 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260820-21:47:04 (UTC+8)
38252026-08-20 21:37:01 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260820-21:37:01 (UTC+8)
38242026-08-20 21:27:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260820-21:27:01 (UTC+8)
38232026-08-20 21:17:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260820-21:17:00 (UTC+8)
38222026-08-20 21:06:59 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260820-21:06:59 (UTC+8)
38212026-08-20 20:56:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-20:56:59 (UTC+8)
38202026-08-20 20:46:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-20:46:58 (UTC+8)
38192026-08-20 20:36:57 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-20:36:58 (UTC+8)
38182026-08-20 20:26:57 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-20:26:57 (UTC+8)
38172026-08-20 20:16:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-20:16:56 (UTC+8)
38162026-08-20 20:06:55 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260820-20:06:56 (UTC+8)
38152026-08-20 19:56:55 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-19:56:55 (UTC+8)
38142026-08-20 19:46:54 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-19:46:55 (UTC+8)
38132026-08-20 19:36:54 done=216/273 running=0 failed=4 pending=53 cuda1:7689/24564MiB 23% cuda3:14/24564MiB 0%260820-19:36:54 (UTC+8)
38122026-08-20 19:26:53 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-19:26:54 (UTC+8)
38112026-08-20 19:16:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-19:16:53 (UTC+8)
38102026-08-20 19:06:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260820-19:06:52 (UTC+8)
38092026-08-20 18:56:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260820-18:56:51 (UTC+8)
38082026-08-20 18:46:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260820-18:46:50 (UTC+8)
38072026-08-20 18:36:49 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260820-18:36:50 (UTC+8)
38062026-08-20 18:26:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260820-18:26:49 (UTC+8)
38052026-08-20 18:16:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260820-18:16:47 (UTC+8)
38042026-08-20 18:06:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-18:06:46 (UTC+8)
38032026-08-20 17:56:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-17:56:45 (UTC+8)
38022026-08-20 17:46:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-17:46:44 (UTC+8)
38012026-08-20 17:36:43 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-17:36:44 (UTC+8)
38002026-08-20 17:26:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-17:26:43 (UTC+8)
37992026-08-20 17:16:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-17:16:42 (UTC+8)
37982026-08-20 17:06:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-17:06:42 (UTC+8)
37972026-08-20 16:56:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-16:56:41 (UTC+8)
37962026-08-20 16:46:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 31% cuda3:14/24564MiB 0%260820-16:46:40 (UTC+8)
37952026-08-20 16:36:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-16:36:40 (UTC+8)
37942026-08-20 16:26:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-16:26:39 (UTC+8)
37932026-08-20 16:16:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260820-16:16:38 (UTC+8)
37922026-08-20 16:06:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260820-16:06:36 (UTC+8)
37912026-08-20 15:56:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-15:56:35 (UTC+8)
37902026-08-20 15:46:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-15:46:35 (UTC+8)
37892026-08-20 15:36:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260820-15:36:34 (UTC+8)
37882026-08-20 15:26:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260820-15:26:32 (UTC+8)
37872026-08-20 15:16:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260820-15:16:31 (UTC+8)
37862026-08-20 15:06:30 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 99% cuda3:14/24564MiB 0%260820-15:06:31 (UTC+8)
37852026-08-20 14:56:29 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 100% cuda3:14/24564MiB 0%260820-14:56:30 (UTC+8)
37842026-08-20 14:46:28 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 99% cuda3:14/24564MiB 0%260820-14:46:29 (UTC+8)
37832026-08-20 14:36:28 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 100% cuda3:14/24564MiB 0%260820-14:36:28 (UTC+8)
37822026-08-20 14:26:27 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 100% cuda3:14/24564MiB 0%260820-14:26:28 (UTC+8)
37812026-08-20 14:16:26 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 100% cuda3:14/24564MiB 0%260820-14:16:27 (UTC+8)
37802026-08-20 14:06:25 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 99% cuda3:14/24564MiB 0%260820-14:06:26 (UTC+8)
37792026-08-20 13:56:22 done=216/273 running=0 failed=4 pending=53 cuda1:13091/24564MiB 94% cuda3:14/24564MiB 0%260820-13:56:25 (UTC+8)
37782026-08-20 13:46:21 done=216/273 running=0 failed=4 pending=53 cuda1:13107/24564MiB 96% cuda3:14/24564MiB 0%260820-13:46:22 (UTC+8)
37772026-08-20 13:36:20 done=216/273 running=0 failed=4 pending=53 cuda1:9561/24564MiB 94% cuda3:14/24564MiB 0%260820-13:36:21 (UTC+8)
37762026-08-20 13:26:20 done=216/273 running=0 failed=4 pending=53 cuda1:1435/24564MiB 72% cuda3:14/24564MiB 0%260820-13:26:20 (UTC+8)
37752026-08-20 13:16:19 done=216/273 running=0 failed=4 pending=53 cuda1:13003/24564MiB 98% cuda3:14/24564MiB 0%260820-13:16:19 (UTC+8)
37742026-08-20 13:06:18 done=216/273 running=0 failed=4 pending=53 cuda1:13107/24564MiB 94% cuda3:14/24564MiB 0%260820-13:06:19 (UTC+8)
37732026-08-20 12:56:18 done=216/273 running=0 failed=4 pending=53 cuda1:13109/24564MiB 94% cuda3:14/24564MiB 0%260820-12:56:18 (UTC+8)
37722026-08-20 12:46:15 done=216/273 running=0 failed=4 pending=53 cuda1:9561/24564MiB 96% cuda3:14/24564MiB 0%260820-12:46:17 (UTC+8)
37712026-08-20 12:36:14 done=216/273 running=0 failed=4 pending=53 cuda1:1637/24564MiB 80% cuda3:14/24564MiB 0%260820-12:36:15 (UTC+8)
37702026-08-20 12:26:13 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260820-12:26:14 (UTC+8)
37692026-08-20 12:16:11 done=216/273 running=0 failed=4 pending=53 cuda1:525/24564MiB 14% cuda3:14/24564MiB 0%260820-12:16:13 (UTC+8)
37682026-08-20 12:06:10 done=216/273 running=0 failed=4 pending=53 cuda1:523/24564MiB 14% cuda3:14/24564MiB 0%260820-12:06:11 (UTC+8)
37672026-08-20 11:56:09 done=216/273 running=0 failed=4 pending=53 cuda1:523/24564MiB 11% cuda3:14/24564MiB 0%260820-11:56:10 (UTC+8)
37662026-08-20 11:46:08 done=216/273 running=0 failed=4 pending=53 cuda1:523/24564MiB 14% cuda3:14/24564MiB 0%260820-11:46:09 (UTC+8)
37652026-08-20 11:36:07 done=216/273 running=0 failed=4 pending=53 cuda1:523/24564MiB 12% cuda3:14/24564MiB 0%260820-11:36:08 (UTC+8)
37642026-08-20 11:26:07 done=216/273 running=0 failed=4 pending=53 cuda1:525/24564MiB 16% cuda3:14/24564MiB 0%260820-11:26:07 (UTC+8)
37632026-08-20 11:16:06 done=216/273 running=0 failed=4 pending=53 cuda1:525/24564MiB 15% cuda3:14/24564MiB 0%260820-11:16:07 (UTC+8)
37622026-08-20 11:06:05 done=216/273 running=0 failed=4 pending=53 cuda1:525/24564MiB 15% cuda3:14/24564MiB 0%260820-11:06:06 (UTC+8)
37612026-08-20 10:56:04 done=216/273 running=0 failed=4 pending=53 cuda1:523/24564MiB 17% cuda3:14/24564MiB 0%260820-10:56:05 (UTC+8)
37602026-08-20 10:46:03 done=216/273 running=0 failed=4 pending=53 cuda1:13109/24564MiB 95% cuda3:14/24564MiB 0%260820-10:46:04 (UTC+8)
37592026-08-20 10:36:02 done=216/273 running=0 failed=4 pending=53 cuda1:13109/24564MiB 96% cuda3:14/24564MiB 0%260820-10:36:03 (UTC+8)
37582026-08-20 10:26:01 done=216/273 running=0 failed=4 pending=53 cuda1:13107/24564MiB 94% cuda3:14/24564MiB 0%260820-10:26:02 (UTC+8)
37572026-08-20 10:16:01 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-10:16:01 (UTC+8)
37562026-08-20 10:06:00 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260820-10:06:01 (UTC+8)
37552026-08-20 09:56:00 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-09:56:00 (UTC+8)
37542026-08-20 09:45:58 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 51% cuda3:14/24564MiB 0%260820-09:45:59 (UTC+8)
37532026-08-20 09:35:58 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-09:35:58 (UTC+8)
37522026-08-20 09:25:57 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260820-09:25:58 (UTC+8)
37512026-08-20 09:15:56 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 49% cuda3:14/24564MiB 0%260820-09:15:57 (UTC+8)
37502026-08-20 09:05:55 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-09:05:56 (UTC+8)
37492026-08-20 08:55:54 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 55% cuda3:14/24564MiB 0%260820-08:55:55 (UTC+8)
37482026-08-20 08:45:54 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-08:45:54 (UTC+8)
37472026-08-20 08:35:53 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260820-08:35:54 (UTC+8)
37462026-08-20 08:25:52 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260820-08:25:53 (UTC+8)
37452026-08-20 08:15:51 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 50% cuda3:14/24564MiB 0%260820-08:15:52 (UTC+8)
37442026-08-20 08:05:50 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 51% cuda3:14/24564MiB 0%260820-08:05:51 (UTC+8)
37432026-08-20 07:55:50 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-07:55:50 (UTC+8)
37422026-08-20 07:45:49 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260820-07:45:49 (UTC+8)
37412026-08-20 07:35:48 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260820-07:35:49 (UTC+8)
37402026-08-20 07:25:48 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-07:25:48 (UTC+8)
37392026-08-20 07:15:47 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260820-07:15:48 (UTC+8)
37382026-08-20 07:05:47 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-07:05:47 (UTC+8)
37372026-08-20 06:55:46 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 85% cuda3:14/24564MiB 0%260820-06:55:46 (UTC+8)
37362026-08-20 06:45:45 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-06:45:46 (UTC+8)
37352026-08-20 06:35:44 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260820-06:35:45 (UTC+8)
37342026-08-20 06:25:44 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 52% cuda3:14/24564MiB 0%260820-06:25:44 (UTC+8)
37332026-08-20 06:15:43 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 52% cuda3:14/24564MiB 0%260820-06:15:44 (UTC+8)
37322026-08-20 06:05:43 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 49% cuda3:14/24564MiB 0%260820-06:05:43 (UTC+8)
37312026-08-20 05:55:42 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260820-05:55:43 (UTC+8)
37302026-08-20 05:45:41 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 51% cuda3:14/24564MiB 0%260820-05:45:42 (UTC+8)
37292026-08-20 05:35:41 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-05:35:41 (UTC+8)
37282026-08-20 05:25:40 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-05:25:41 (UTC+8)
37272026-08-20 05:15:40 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-05:15:40 (UTC+8)
37262026-08-20 05:05:39 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-05:05:39 (UTC+8)
37252026-08-20 04:55:38 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-04:55:39 (UTC+8)
37242026-08-20 04:45:37 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260820-04:45:38 (UTC+8)
37232026-08-20 04:35:36 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-04:35:37 (UTC+8)
37222026-08-20 04:25:36 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-04:25:36 (UTC+8)
37212026-08-20 04:15:35 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 50% cuda3:14/24564MiB 0%260820-04:15:36 (UTC+8)
37202026-08-20 04:05:34 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-04:05:35 (UTC+8)
37192026-08-20 03:55:34 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-03:55:34 (UTC+8)
37182026-08-20 03:45:33 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 50% cuda3:14/24564MiB 0%260820-03:45:34 (UTC+8)
37172026-08-20 03:35:32 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 51% cuda3:14/24564MiB 0%260820-03:35:33 (UTC+8)
37162026-08-20 03:25:31 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-03:25:32 (UTC+8)
37152026-08-20 03:15:31 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260820-03:15:31 (UTC+8)
37142026-08-20 03:05:30 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 51% cuda3:14/24564MiB 0%260820-03:05:31 (UTC+8)
37132026-08-20 02:55:29 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 52% cuda3:14/24564MiB 0%260820-02:55:30 (UTC+8)
37122026-08-20 02:45:29 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 52% cuda3:14/24564MiB 0%260820-02:45:29 (UTC+8)
37112026-08-20 02:35:28 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-02:35:29 (UTC+8)
37102026-08-20 02:25:27 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260820-02:25:28 (UTC+8)
37092026-08-20 02:15:27 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 51% cuda3:14/24564MiB 0%260820-02:15:27 (UTC+8)
37082026-08-20 02:05:26 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 46% cuda3:14/24564MiB 0%260820-02:05:27 (UTC+8)
37072026-08-20 01:55:25 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 52% cuda3:14/24564MiB 0%260820-01:55:26 (UTC+8)
37062026-08-20 01:45:24 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260820-01:45:25 (UTC+8)
37052026-08-20 01:35:23 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-01:35:24 (UTC+8)
37042026-08-20 01:25:23 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-01:25:23 (UTC+8)
37032026-08-20 01:15:22 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-01:15:23 (UTC+8)
37022026-08-20 01:05:22 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260820-01:05:22 (UTC+8)
37012026-08-20 00:55:21 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-00:55:22 (UTC+8)
37002026-08-20 00:45:20 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260820-00:45:21 (UTC+8)
36992026-08-20 00:35:19 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-00:35:20 (UTC+8)
36982026-08-20 00:25:19 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-00:25:19 (UTC+8)
36972026-08-20 00:15:18 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-00:15:19 (UTC+8)
36962026-08-20 00:05:17 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260820-00:05:18 (UTC+8)
36952026-08-19 23:55:17 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 50% cuda3:14/24564MiB 0%260819-23:55:17 (UTC+8)
36942026-08-19 23:45:15 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260819-23:45:17 (UTC+8)
36932026-08-19 23:35:14 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260819-23:35:15 (UTC+8)
36922026-08-19 23:25:14 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260819-23:25:14 (UTC+8)
36912026-08-19 23:15:13 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260819-23:15:14 (UTC+8)
36902026-08-19 23:05:13 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260819-23:05:13 (UTC+8)
36892026-08-19 22:55:12 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260819-22:55:12 (UTC+8)
36882026-08-19 22:45:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 94% cuda3:14/24564MiB 0%260819-22:45:12 (UTC+8)
36872026-08-19 22:35:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-22:35:11 (UTC+8)
36862026-08-19 22:25:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-22:25:11 (UTC+8)
36852026-08-19 22:15:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-22:15:10 (UTC+8)
36842026-08-19 22:05:09 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-22:05:10 (UTC+8)
36832026-08-19 21:55:08 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-21:55:09 (UTC+8)
36822026-08-19 21:45:07 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-21:45:08 (UTC+8)
36812026-08-19 21:35:06 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-21:35:06 (UTC+8)
36802026-08-19 21:25:05 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-21:25:06 (UTC+8)
36792026-08-19 21:15:05 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-21:15:05 (UTC+8)
36782026-08-19 21:05:04 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-21:05:05 (UTC+8)
36772026-08-19 20:55:03 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-20:55:04 (UTC+8)
36762026-08-19 20:45:02 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-20:45:03 (UTC+8)
36752026-08-19 20:35:02 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 99% cuda3:14/24564MiB 0%260819-20:35:02 (UTC+8)
36742026-08-19 20:25:01 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260819-20:25:02 (UTC+8)
36732026-08-19 20:15:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-20:15:01 (UTC+8)
36722026-08-19 20:04:59 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-20:05:00 (UTC+8)
36712026-08-19 19:54:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-19:54:59 (UTC+8)
36702026-08-19 19:44:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-19:44:58 (UTC+8)
36692026-08-19 19:34:57 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-19:34:58 (UTC+8)
36682026-08-19 19:24:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-19:24:57 (UTC+8)
36672026-08-19 19:14:55 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-19:14:56 (UTC+8)
36662026-08-19 19:04:54 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-19:04:55 (UTC+8)
36652026-08-19 18:54:53 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-18:54:54 (UTC+8)
36642026-08-19 18:44:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-18:44:53 (UTC+8)
36632026-08-19 18:34:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-18:34:52 (UTC+8)
36622026-08-19 18:24:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-18:24:52 (UTC+8)
36612026-08-19 18:14:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-18:14:51 (UTC+8)
36602026-08-19 18:04:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260819-18:04:51 (UTC+8)
36592026-08-19 17:54:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-17:54:50 (UTC+8)
36582026-08-19 17:44:49 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-17:44:50 (UTC+8)
36572026-08-19 17:34:48 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-17:34:48 (UTC+8)
36562026-08-19 17:24:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260819-17:24:48 (UTC+8)
36552026-08-19 17:14:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-17:14:47 (UTC+8)
36542026-08-19 17:04:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-17:04:47 (UTC+8)
36532026-08-19 16:54:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-16:54:46 (UTC+8)
36522026-08-19 16:44:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-16:44:45 (UTC+8)
36512026-08-19 16:34:43 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-16:34:44 (UTC+8)
36502026-08-19 16:24:43 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-16:24:43 (UTC+8)
36492026-08-19 16:14:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-16:14:42 (UTC+8)
36482026-08-19 16:04:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-16:04:42 (UTC+8)
36472026-08-19 15:54:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-15:54:41 (UTC+8)
36462026-08-19 15:44:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-15:44:40 (UTC+8)
36452026-08-19 15:34:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-15:34:39 (UTC+8)
36442026-08-19 15:24:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-15:24:39 (UTC+8)
36432026-08-19 15:14:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-15:14:38 (UTC+8)
36422026-08-19 15:04:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-15:04:37 (UTC+8)
36412026-08-19 14:54:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 74% cuda3:14/24564MiB 0%260819-14:54:37 (UTC+8)
36402026-08-19 14:44:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1917/24564MiB 49%260819-14:44:36 (UTC+8)
36392026-08-19 14:34:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1917/24564MiB 48%260819-14:34:36 (UTC+8)
36382026-08-19 14:24:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1917/24564MiB 45%260819-14:24:35 (UTC+8)
36372026-08-19 14:14:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1917/24564MiB 63%260819-14:14:34 (UTC+8)
36362026-08-19 14:04:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 66%260819-14:04:33 (UTC+8)
36352026-08-19 13:54:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 47%260819-13:54:33 (UTC+8)
36342026-08-19 13:44:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:1915/24564MiB 48%260819-13:44:32 (UTC+8)
36332026-08-19 13:34:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:1915/24564MiB 30%260819-13:34:32 (UTC+8)
36322026-08-19 13:24:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:1915/24564MiB 16%260819-13:24:31 (UTC+8)
36312026-08-19 13:14:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:1915/24564MiB 47%260819-13:14:31 (UTC+8)
36302026-08-19 13:04:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 50%260819-13:04:30 (UTC+8)
36292026-08-19 12:54:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:1915/24564MiB 47%260819-12:54:29 (UTC+8)
36282026-08-19 12:44:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 47%260819-12:44:28 (UTC+8)
36272026-08-19 12:34:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 49%260819-12:34:26 (UTC+8)
36262026-08-19 12:24:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:1915/24564MiB 49%260819-12:24:26 (UTC+8)
36252026-08-19 12:14:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 49%260819-12:14:25 (UTC+8)
36242026-08-19 12:04:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 48%260819-12:04:25 (UTC+8)
36232026-08-19 11:54:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 14%260819-11:54:24 (UTC+8)
36222026-08-19 11:44:23 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:1915/24564MiB 49%260819-11:44:23 (UTC+8)
36212026-08-19 11:34:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 47%260819-11:34:23 (UTC+8)
36202026-08-19 11:24:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 47%260819-11:24:22 (UTC+8)
36192026-08-19 11:14:21 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 68% cuda3:1915/24564MiB 47%260819-11:14:22 (UTC+8)
36182026-08-19 11:04:20 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 47%260819-11:04:21 (UTC+8)
36172026-08-19 10:54:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 50%260819-10:54:20 (UTC+8)
36162026-08-19 10:44:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 66%260819-10:44:19 (UTC+8)
36152026-08-19 10:34:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:1915/24564MiB 49%260819-10:34:19 (UTC+8)
36142026-08-19 10:24:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1915/24564MiB 50%260819-10:24:18 (UTC+8)
36132026-08-19 10:14:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-10:14:17 (UTC+8)
36122026-08-19 10:04:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-10:04:16 (UTC+8)
36112026-08-19 09:54:15 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-09:54:16 (UTC+8)
36102026-08-19 09:44:15 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-09:44:15 (UTC+8)
36092026-08-19 09:34:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-09:34:15 (UTC+8)
36082026-08-19 09:24:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-09:24:14 (UTC+8)
36072026-08-19 09:14:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-09:14:13 (UTC+8)
36062026-08-19 09:04:12 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-09:04:13 (UTC+8)
36052026-08-19 08:54:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-08:54:12 (UTC+8)
36042026-08-19 08:44:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-08:44:11 (UTC+8)
36032026-08-19 08:34:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-08:34:10 (UTC+8)
36022026-08-19 08:24:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-08:24:09 (UTC+8)
36012026-08-19 08:14:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-08:14:08 (UTC+8)
36002026-08-19 08:04:07 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-08:04:08 (UTC+8)
35992026-08-19 07:54:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-07:54:07 (UTC+8)
35982026-08-19 07:44:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-07:44:06 (UTC+8)
35972026-08-19 07:34:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260819-07:34:06 (UTC+8)
35962026-08-19 07:24:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260819-07:24:05 (UTC+8)
35952026-08-19 07:14:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260819-07:14:05 (UTC+8)
35942026-08-19 07:04:03 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-07:04:04 (UTC+8)
35932026-08-19 06:54:02 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-06:54:03 (UTC+8)
35922026-08-19 06:44:01 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-06:44:02 (UTC+8)
35912026-08-19 06:34:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260819-06:34:01 (UTC+8)
35902026-08-19 06:24:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-06:24:00 (UTC+8)
35892026-08-19 06:13:59 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-06:14:00 (UTC+8)
35882026-08-19 06:03:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-06:03:59 (UTC+8)
35872026-08-19 05:53:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-05:53:58 (UTC+8)
35862026-08-19 05:43:57 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-05:43:57 (UTC+8)
35852026-08-19 05:33:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-05:33:57 (UTC+8)
35842026-08-19 05:23:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-05:23:56 (UTC+8)
35832026-08-19 05:13:55 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-05:13:56 (UTC+8)
35822026-08-19 05:03:55 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-05:03:55 (UTC+8)
35812026-08-19 04:53:54 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-04:53:55 (UTC+8)
35802026-08-19 04:43:54 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-04:43:54 (UTC+8)
35792026-08-19 04:33:53 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260819-04:33:53 (UTC+8)
35782026-08-19 04:23:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-04:23:53 (UTC+8)
35772026-08-19 04:13:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-04:13:52 (UTC+8)
35762026-08-19 04:03:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-04:03:51 (UTC+8)
35752026-08-19 03:53:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260819-03:53:51 (UTC+8)
35742026-08-19 03:43:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-03:43:50 (UTC+8)
35732026-08-19 03:33:49 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-03:33:50 (UTC+8)
35722026-08-19 03:23:48 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-03:23:49 (UTC+8)
35712026-08-19 03:13:48 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-03:13:48 (UTC+8)
35702026-08-19 03:03:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-03:03:47 (UTC+8)
35692026-08-19 02:53:46 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:14/24564MiB 0%260819-02:53:47 (UTC+8)
35682026-08-19 02:43:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-02:43:46 (UTC+8)
35672026-08-19 02:33:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-02:33:46 (UTC+8)
35662026-08-19 02:23:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-02:23:45 (UTC+8)
35652026-08-19 02:13:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260819-02:13:45 (UTC+8)
35642026-08-19 02:03:43 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-02:03:44 (UTC+8)
35632026-08-19 01:53:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-01:53:43 (UTC+8)
35622026-08-19 01:43:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260819-01:43:42 (UTC+8)
35612026-08-19 01:33:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260819-01:33:41 (UTC+8)
35602026-08-19 01:23:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-01:23:41 (UTC+8)
35592026-08-19 01:13:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-01:13:40 (UTC+8)
35582026-08-19 01:03:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-01:03:39 (UTC+8)
35572026-08-19 00:53:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260819-00:53:39 (UTC+8)
35562026-08-19 00:43:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-00:43:37 (UTC+8)
35552026-08-19 00:33:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-00:33:36 (UTC+8)
35542026-08-19 00:23:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260819-00:23:36 (UTC+8)
35532026-08-19 00:13:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-00:13:35 (UTC+8)
35522026-08-19 00:03:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260819-00:03:34 (UTC+8)
35512026-08-18 23:53:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260818-23:53:34 (UTC+8)
35502026-08-18 23:43:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 89% cuda3:14/24564MiB 0%260818-23:43:33 (UTC+8)
35492026-08-18 23:33:31 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 99% cuda3:14/24564MiB 0%260818-23:33:32 (UTC+8)
35482026-08-18 23:23:30 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 100% cuda3:14/24564MiB 0%260818-23:23:31 (UTC+8)
35472026-08-18 23:13:30 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 99% cuda3:14/24564MiB 0%260818-23:13:30 (UTC+8)
35462026-08-18 23:03:28 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 99% cuda3:14/24564MiB 0%260818-23:03:29 (UTC+8)
35452026-08-18 22:53:27 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 99% cuda3:14/24564MiB 0%260818-22:53:28 (UTC+8)
35442026-08-18 22:43:26 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 100% cuda3:14/24564MiB 0%260818-22:43:27 (UTC+8)
35432026-08-18 22:33:25 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 100% cuda3:14/24564MiB 0%260818-22:33:26 (UTC+8)
35422026-08-18 22:23:25 done=216/273 running=0 failed=4 pending=53 cuda1:13111/24564MiB 94% cuda3:14/24564MiB 0%260818-22:23:25 (UTC+8)
35412026-08-18 22:13:24 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:14/24564MiB 0%260818-22:13:25 (UTC+8)
35402026-08-18 22:03:24 done=216/273 running=0 failed=4 pending=53 cuda1:1201/24564MiB 74% cuda3:14/24564MiB 0%260818-22:03:24 (UTC+8)
35392026-08-18 21:53:23 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:14/24564MiB 0%260818-21:53:24 (UTC+8)
35382026-08-18 21:43:22 done=216/273 running=0 failed=4 pending=53 cuda1:1111/24564MiB 83% cuda3:14/24564MiB 0%260818-21:43:23 (UTC+8)
35372026-08-18 21:33:22 done=216/273 running=0 failed=4 pending=53 cuda1:13109/24564MiB 96% cuda3:14/24564MiB 0%260818-21:33:22 (UTC+8)
35362026-08-18 21:23:21 done=216/273 running=0 failed=4 pending=53 cuda1:7693/24564MiB 92% cuda3:14/24564MiB 0%260818-21:23:21 (UTC+8)
35352026-08-18 21:13:20 done=216/273 running=0 failed=4 pending=53 cuda1:1553/24564MiB 78% cuda3:14/24564MiB 0%260818-21:13:20 (UTC+8)
35342026-08-18 21:03:19 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-21:03:19 (UTC+8)
35332026-08-18 20:53:18 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 55% cuda3:14/24564MiB 0%260818-20:53:19 (UTC+8)
35322026-08-18 20:43:18 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 52% cuda3:14/24564MiB 0%260818-20:43:18 (UTC+8)
35312026-08-18 20:33:17 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-20:33:17 (UTC+8)
35302026-08-18 20:23:16 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 57% cuda3:14/24564MiB 0%260818-20:23:17 (UTC+8)
35292026-08-18 20:13:15 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 51% cuda3:14/24564MiB 0%260818-20:13:16 (UTC+8)
35282026-08-18 20:03:14 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-20:03:15 (UTC+8)
35272026-08-18 19:53:13 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-19:53:14 (UTC+8)
35262026-08-18 19:43:12 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-19:43:12 (UTC+8)
35252026-08-18 19:33:11 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 51% cuda3:14/24564MiB 0%260818-19:33:12 (UTC+8)
35242026-08-18 19:23:10 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260818-19:23:11 (UTC+8)
35232026-08-18 19:13:10 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-19:13:10 (UTC+8)
35222026-08-18 19:03:09 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-19:03:10 (UTC+8)
35212026-08-18 18:53:08 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260818-18:53:09 (UTC+8)
35202026-08-18 18:43:08 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 52% cuda3:14/24564MiB 0%260818-18:43:08 (UTC+8)
35192026-08-18 18:33:07 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-18:33:08 (UTC+8)
35182026-08-18 18:23:06 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-18:23:07 (UTC+8)
35172026-08-18 18:13:05 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 51% cuda3:14/24564MiB 0%260818-18:13:06 (UTC+8)
35162026-08-18 18:03:04 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-18:03:05 (UTC+8)
35152026-08-18 17:53:04 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-17:53:04 (UTC+8)
35142026-08-18 17:43:03 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-17:43:03 (UTC+8)
35132026-08-18 17:33:01 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-17:33:03 (UTC+8)
35122026-08-18 17:23:00 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-17:23:01 (UTC+8)
35112026-08-18 17:12:59 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 51% cuda3:14/24564MiB 0%260818-17:13:00 (UTC+8)
35102026-08-18 17:02:58 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 54% cuda3:14/24564MiB 0%260818-17:02:59 (UTC+8)
35092026-08-18 16:52:57 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 53% cuda3:14/24564MiB 0%260818-16:52:58 (UTC+8)
35082026-08-18 16:42:57 done=216/273 running=0 failed=4 pending=53 cuda1:7817/24564MiB 51% cuda3:14/24564MiB 0%260818-16:42:57 (UTC+8)
35072026-08-18 16:32:56 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-16:32:57 (UTC+8)
35062026-08-18 16:22:55 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 55% cuda3:14/24564MiB 0%260818-16:22:56 (UTC+8)
35052026-08-18 16:02:46 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 53% cuda3:14/24564MiB 0%260818-16:02:47 (UTC+8)
35042026-08-18 15:52:46 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 51% cuda3:14/24564MiB 0%260818-15:52:46 (UTC+8)
35032026-08-18 15:42:45 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-15:42:46 (UTC+8)
35022026-08-18 14:52:13 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 55% cuda3:14/24564MiB 0%260818-14:52:13 (UTC+8)
35012026-08-18 14:42:12 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 85% cuda3:14/24564MiB 0%260818-14:42:12 (UTC+8)
35002026-08-18 14:32:10 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-14:32:11 (UTC+8)
34992026-08-18 14:22:09 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 65% cuda3:14/24564MiB 0%260818-14:22:10 (UTC+8)
34982026-08-18 14:12:09 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-14:12:09 (UTC+8)
34972026-08-18 14:02:08 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 50% cuda3:14/24564MiB 0%260818-14:02:09 (UTC+8)
34962026-08-18 13:52:07 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-13:52:08 (UTC+8)
34952026-08-18 13:42:07 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-13:42:07 (UTC+8)
34942026-08-18 13:32:06 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 51% cuda3:14/24564MiB 0%260818-13:32:06 (UTC+8)
34932026-08-18 13:22:05 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 71% cuda3:14/24564MiB 0%260818-13:22:06 (UTC+8)
34922026-08-18 13:12:05 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-13:12:05 (UTC+8)
34912026-08-18 13:02:04 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 51% cuda3:14/24564MiB 0%260818-13:02:05 (UTC+8)
34902026-08-18 12:52:03 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 52% cuda3:14/24564MiB 0%260818-12:52:04 (UTC+8)
34892026-08-18 12:42:03 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 50% cuda3:14/24564MiB 0%260818-12:42:03 (UTC+8)
34882026-08-18 12:32:02 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-12:32:03 (UTC+8)
34872026-08-18 12:22:02 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-12:22:02 (UTC+8)
34862026-08-18 12:12:01 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-12:12:02 (UTC+8)
34852026-08-18 12:02:00 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-12:02:01 (UTC+8)
34842026-08-18 11:51:59 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-11:52:00 (UTC+8)
34832026-08-18 11:41:58 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-11:41:59 (UTC+8)
34822026-08-18 11:31:58 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 51% cuda3:14/24564MiB 0%260818-11:31:58 (UTC+8)
34812026-08-18 11:21:57 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-11:21:58 (UTC+8)
34802026-08-18 11:11:56 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-11:11:57 (UTC+8)
34792026-08-18 11:01:55 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 50% cuda3:14/24564MiB 0%260818-11:01:56 (UTC+8)
34782026-08-18 10:51:55 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-10:51:55 (UTC+8)
34772026-08-18 10:41:54 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-10:41:55 (UTC+8)
34762026-08-18 10:31:53 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 54% cuda3:14/24564MiB 0%260818-10:31:54 (UTC+8)
34752026-08-18 10:21:53 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 47% cuda3:14/24564MiB 0%260818-10:21:53 (UTC+8)
34742026-08-18 10:11:52 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 49% cuda3:14/24564MiB 0%260818-10:11:52 (UTC+8)
34732026-08-18 10:01:51 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 48% cuda3:14/24564MiB 0%260818-10:01:51 (UTC+8)
34722026-08-18 09:51:50 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 40% cuda3:14/24564MiB 0%260818-09:51:50 (UTC+8)
34712026-08-18 09:41:49 done=216/273 running=0 failed=4 pending=53 cuda1:7751/24564MiB 47% cuda3:14/24564MiB 0%260818-09:41:50 (UTC+8)
34702026-08-18 09:31:49 done=216/273 running=0 failed=4 pending=53 cuda1:7731/24564MiB 32% cuda3:14/24564MiB 0%260818-09:31:49 (UTC+8)
34692026-08-18 09:21:48 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260818-09:21:48 (UTC+8)
34682026-08-18 09:11:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-09:11:47 (UTC+8)
34672026-08-18 09:01:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260818-09:01:47 (UTC+8)
34662026-08-18 08:51:45 done=216/273 running=0 failed=4 pending=53 cuda1:7835/24564MiB 0% cuda3:14/24564MiB 0%260818-08:51:46 (UTC+8)
34652026-08-18 08:41:44 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260818-08:41:45 (UTC+8)
34642026-08-18 08:31:43 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 98% cuda3:14/24564MiB 0%260818-08:31:44 (UTC+8)
34632026-08-18 08:21:43 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260818-08:21:43 (UTC+8)
34622026-08-18 08:11:42 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260818-08:11:42 (UTC+8)
34612026-08-18 08:01:41 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260818-08:01:42 (UTC+8)
34602026-08-18 07:51:41 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260818-07:51:41 (UTC+8)
34592026-08-18 07:41:40 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260818-07:41:41 (UTC+8)
34582026-08-18 07:31:39 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260818-07:31:40 (UTC+8)
34572026-08-18 07:21:38 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260818-07:21:39 (UTC+8)
34562026-08-18 07:11:38 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:14/24564MiB 0%260818-07:11:38 (UTC+8)
34552026-08-18 07:01:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-07:01:38 (UTC+8)
34542026-08-18 06:51:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-06:51:37 (UTC+8)
34532026-08-18 06:41:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-06:41:37 (UTC+8)
34522026-08-18 06:31:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260818-06:31:36 (UTC+8)
34512026-08-18 06:21:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260818-06:21:35 (UTC+8)
34502026-08-18 06:11:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260818-06:11:34 (UTC+8)
34492026-08-18 06:01:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260818-06:01:34 (UTC+8)
34482026-08-18 05:51:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-05:51:33 (UTC+8)
34472026-08-18 05:41:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260818-05:41:32 (UTC+8)
34462026-08-18 05:31:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-05:31:31 (UTC+8)
34452026-08-18 05:21:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260818-05:21:31 (UTC+8)
34442026-08-18 05:11:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260818-05:11:30 (UTC+8)
34432026-08-18 05:01:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260818-05:01:30 (UTC+8)
34422026-08-18 04:51:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260818-04:51:29 (UTC+8)
34412026-08-18 04:41:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260818-04:41:28 (UTC+8)
34402026-08-18 04:31:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260818-04:31:28 (UTC+8)
34392026-08-18 04:21:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260818-04:21:27 (UTC+8)
34382026-08-18 04:11:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260818-04:11:26 (UTC+8)
34372026-08-18 04:01:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-04:01:26 (UTC+8)
34362026-08-18 03:51:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-03:51:25 (UTC+8)
34352026-08-18 03:41:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-03:41:24 (UTC+8)
34342026-08-18 03:31:23 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-03:31:24 (UTC+8)
34332026-08-18 03:21:23 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 94% cuda3:14/24564MiB 0%260818-03:21:23 (UTC+8)
34322026-08-18 03:11:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-03:11:23 (UTC+8)
34312026-08-18 03:01:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-03:01:22 (UTC+8)
34302026-08-18 02:51:21 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:14/24564MiB 0%260818-02:51:21 (UTC+8)
34292026-08-18 02:41:20 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-02:41:21 (UTC+8)
34282026-08-18 02:31:20 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-02:31:20 (UTC+8)
34272026-08-18 02:21:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-02:21:20 (UTC+8)
34262026-08-18 02:11:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-02:11:19 (UTC+8)
34252026-08-18 02:01:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-02:01:19 (UTC+8)
34242026-08-18 01:51:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-01:51:18 (UTC+8)
34232026-08-18 01:41:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-01:41:18 (UTC+8)
34222026-08-18 01:31:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-01:31:17 (UTC+8)
34212026-08-18 01:21:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 56% cuda3:14/24564MiB 0%260818-01:21:16 (UTC+8)
34202026-08-18 01:11:15 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 99% cuda3:14/24564MiB 0%260818-01:11:16 (UTC+8)
34192026-08-18 01:01:15 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260818-01:01:15 (UTC+8)
34182026-08-18 00:51:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-00:51:15 (UTC+8)
34172026-08-18 00:41:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260818-00:41:14 (UTC+8)
34162026-08-18 00:31:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:14/24564MiB 0%260818-00:31:14 (UTC+8)
34152026-08-18 00:21:12 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-00:21:13 (UTC+8)
34142026-08-18 00:11:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:14/24564MiB 0%260818-00:11:12 (UTC+8)
34132026-08-18 00:01:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260818-00:01:10 (UTC+8)
34122026-08-17 23:51:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260817-23:51:09 (UTC+8)
34112026-08-17 23:41:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260817-23:41:09 (UTC+8)
34102026-08-17 23:31:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:14/24564MiB 0%260817-23:31:08 (UTC+8)
34092026-08-17 23:21:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:14/24564MiB 0%260817-23:21:06 (UTC+8)
34082026-08-17 23:11:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:14/24564MiB 0%260817-23:11:06 (UTC+8)
34072026-08-17 23:01:04 done=216/273 running=0 failed=4 pending=53 cuda1:8491/24564MiB 98% cuda3:14/24564MiB 0%260817-23:01:05 (UTC+8)
34062026-08-17 22:51:04 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 99% cuda3:14/24564MiB 0%260817-22:51:04 (UTC+8)
34052026-08-17 22:41:03 done=216/273 running=0 failed=4 pending=53 cuda1:8473/24564MiB 13% cuda3:14/24564MiB 0%260817-22:41:03 (UTC+8)
34042026-08-17 22:31:02 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 98% cuda3:14/24564MiB 0%260817-22:31:02 (UTC+8)
34032026-08-17 22:21:01 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 14% cuda3:14/24564MiB 0%260817-22:21:02 (UTC+8)
34022026-08-17 22:11:01 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 98% cuda3:14/24564MiB 0%260817-22:11:01 (UTC+8)
34012026-08-17 22:01:00 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 15% cuda3:14/24564MiB 0%260817-22:01:01 (UTC+8)
34002026-08-17 21:51:00 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 98% cuda3:14/24564MiB 0%260817-21:51:00 (UTC+8)
33992026-08-17 21:40:59 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 14% cuda3:14/24564MiB 0%260817-21:41:00 (UTC+8)
33982026-08-17 21:30:58 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 99% cuda3:14/24564MiB 0%260817-21:30:59 (UTC+8)
33972026-08-17 21:20:58 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 14% cuda3:14/24564MiB 0%260817-21:20:58 (UTC+8)
33962026-08-17 21:10:57 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 14% cuda3:14/24564MiB 0%260817-21:10:58 (UTC+8)
33952026-08-17 21:00:56 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 13% cuda3:14/24564MiB 0%260817-21:00:57 (UTC+8)
33942026-08-17 20:50:56 done=216/273 running=0 failed=4 pending=53 cuda1:8487/24564MiB 99% cuda3:14/24564MiB 0%260817-20:50:56 (UTC+8)
33932026-08-17 20:40:55 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 97% cuda3:14/24564MiB 0%260817-20:40:55 (UTC+8)
33922026-08-17 20:30:54 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 14% cuda3:14/24564MiB 0%260817-20:30:55 (UTC+8)
33912026-08-17 20:20:54 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 14% cuda3:14/24564MiB 0%260817-20:20:54 (UTC+8)
33902026-08-17 20:10:53 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 14% cuda3:14/24564MiB 0%260817-20:10:53 (UTC+8)
33892026-08-17 20:00:52 done=216/273 running=0 failed=4 pending=53 cuda1:8495/24564MiB 14% cuda3:14/24564MiB 0%260817-20:00:53 (UTC+8)
33882026-08-17 19:50:51 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 100% cuda3:14/24564MiB 0%260817-19:50:52 (UTC+8)
33872026-08-17 19:40:50 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 96% cuda3:14/24564MiB 0%260817-19:40:51 (UTC+8)
33862026-08-17 19:30:50 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 100% cuda3:14/24564MiB 0%260817-19:30:50 (UTC+8)
33852026-08-17 19:20:49 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 100% cuda3:14/24564MiB 0%260817-19:20:50 (UTC+8)
33842026-08-17 19:10:49 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 97% cuda3:14/24564MiB 0%260817-19:10:49 (UTC+8)
33832026-08-17 19:00:48 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 97% cuda3:14/24564MiB 0%260817-19:00:49 (UTC+8)
33822026-08-17 18:50:47 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 97% cuda3:14/24564MiB 0%260817-18:50:48 (UTC+8)
33812026-08-17 18:40:46 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 97% cuda3:14/24564MiB 0%260817-18:40:47 (UTC+8)
33802026-08-17 18:30:45 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 98% cuda3:14/24564MiB 0%260817-18:30:46 (UTC+8)
33792026-08-17 18:20:44 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 100% cuda3:14/24564MiB 0%260817-18:20:45 (UTC+8)
33782026-08-17 18:10:43 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 98% cuda3:14/24564MiB 0%260817-18:10:44 (UTC+8)
33772026-08-17 18:00:42 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 98% cuda3:14/24564MiB 0%260817-18:00:43 (UTC+8)
33762026-08-17 17:50:42 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 100% cuda3:14/24564MiB 0%260817-17:50:42 (UTC+8)
33752026-08-17 17:40:41 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 98% cuda3:14/24564MiB 0%260817-17:40:41 (UTC+8)
33742026-08-17 17:30:40 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 48% cuda3:14/24564MiB 0%260817-17:30:41 (UTC+8)
33732026-08-17 17:20:40 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 99% cuda3:14/24564MiB 0%260817-17:20:40 (UTC+8)
33722026-08-17 17:10:38 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 99% cuda3:14/24564MiB 0%260817-17:10:39 (UTC+8)
33712026-08-17 17:00:38 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 97% cuda3:14/24564MiB 0%260817-17:00:38 (UTC+8)
33702026-08-17 16:50:36 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 86% cuda3:14/24564MiB 0%260817-16:50:38 (UTC+8)
33692026-08-17 16:40:35 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:14/24564MiB 0%260817-16:40:36 (UTC+8)
33682026-08-17 16:30:34 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:14/24564MiB 0%260817-16:30:35 (UTC+8)
33672026-08-17 16:20:32 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 15% cuda3:14/24564MiB 0%260817-16:20:34 (UTC+8)
33662026-08-17 16:10:31 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 97% cuda3:14/24564MiB 0%260817-16:10:32 (UTC+8)
33652026-08-17 16:00:31 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:14/24564MiB 0%260817-16:00:31 (UTC+8)
33642026-08-17 15:50:30 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:14/24564MiB 0%260817-15:50:31 (UTC+8)
33632026-08-17 15:40:29 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:14/24564MiB 0%260817-15:40:30 (UTC+8)
33622026-08-17 15:30:29 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 99% cuda3:14/24564MiB 0%260817-15:30:29 (UTC+8)
33612026-08-17 15:20:28 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 99% cuda3:14/24564MiB 0%260817-15:20:29 (UTC+8)
33602026-08-17 15:10:27 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:14/24564MiB 0%260817-15:10:28 (UTC+8)
33592026-08-17 15:00:26 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:14/24564MiB 0%260817-15:00:27 (UTC+8)
33582026-08-17 14:50:26 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:14/24564MiB 0%260817-14:50:26 (UTC+8)
33572026-08-17 14:40:25 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:14/24564MiB 0%260817-14:40:26 (UTC+8)
33562026-08-17 14:30:24 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:14/24564MiB 0%260817-14:30:25 (UTC+8)
33552026-08-17 14:20:22 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:14/24564MiB 0%260817-14:20:24 (UTC+8)
33542026-08-17 14:10:21 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:14/24564MiB 0%260817-14:10:21 (UTC+8)
33532026-08-17 14:00:20 done=216/273 running=0 failed=4 pending=53 cuda1:8353/24564MiB 35% cuda3:14/24564MiB 0%260817-14:00:21 (UTC+8)
33522026-08-17 13:50:20 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 99% cuda3:14/24564MiB 0%260817-13:50:20 (UTC+8)
33512026-08-17 13:40:19 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 99% cuda3:14/24564MiB 0%260817-13:40:20 (UTC+8)
33502026-08-17 13:30:19 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 99% cuda3:14/24564MiB 0%260817-13:30:19 (UTC+8)
33492026-08-17 13:20:18 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:14/24564MiB 0%260817-13:20:19 (UTC+8)
33482026-08-17 13:10:17 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 96% cuda3:14/24564MiB 0%260817-13:10:18 (UTC+8)
33472026-08-17 13:00:16 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:14/24564MiB 0%260817-13:00:17 (UTC+8)
33462026-08-17 12:50:16 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 96% cuda3:1783/24564MiB 14%260817-12:50:16 (UTC+8)
33452026-08-17 12:40:15 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:1783/24564MiB 13%260817-12:40:16 (UTC+8)
33442026-08-17 12:30:14 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 97% cuda3:1783/24564MiB 22%260817-12:30:15 (UTC+8)
33432026-08-17 12:20:14 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:1783/24564MiB 22%260817-12:20:14 (UTC+8)
33422026-08-17 12:10:13 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:1783/24564MiB 23%260817-12:10:13 (UTC+8)
33412026-08-17 12:00:12 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:1783/24564MiB 26%260817-12:00:13 (UTC+8)
33402026-08-17 11:50:11 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 15%260817-11:50:12 (UTC+8)
33392026-08-17 11:40:11 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 22%260817-11:40:11 (UTC+8)
33382026-08-17 11:30:10 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 23%260817-11:30:11 (UTC+8)
33372026-08-17 11:20:09 done=216/273 running=0 failed=4 pending=53 cuda1:8329/24564MiB 37% cuda3:1783/24564MiB 22%260817-11:20:10 (UTC+8)
33362026-08-17 11:10:09 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 99% cuda3:1783/24564MiB 22%260817-11:10:09 (UTC+8)
33352026-08-17 11:00:08 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 99% cuda3:1783/24564MiB 21%260817-11:00:09 (UTC+8)
33342026-08-17 10:50:07 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 99% cuda3:1783/24564MiB 14%260817-10:50:08 (UTC+8)
33332026-08-17 10:40:07 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 22%260817-10:40:07 (UTC+8)
33322026-08-17 10:30:06 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 23%260817-10:30:06 (UTC+8)
33312026-08-17 10:20:04 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 21%260817-10:20:06 (UTC+8)
33302026-08-17 10:10:03 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 64% cuda3:1783/24564MiB 23%260817-10:10:04 (UTC+8)
33292026-08-17 10:00:03 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 15%260817-10:00:03 (UTC+8)
33282026-08-17 09:50:02 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 25%260817-09:50:03 (UTC+8)
33272026-08-17 09:40:01 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 44% cuda3:1783/24564MiB 23%260817-09:40:02 (UTC+8)
33262026-08-17 09:30:00 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 22%260817-09:30:01 (UTC+8)
33252026-08-17 09:19:59 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 100% cuda3:1783/24564MiB 21%260817-09:20:00 (UTC+8)
33242026-08-17 09:09:59 done=216/273 running=0 failed=4 pending=53 cuda1:8513/24564MiB 98% cuda3:1783/24564MiB 22%260817-09:09:59 (UTC+8)
33232026-08-17 08:59:58 done=216/273 running=0 failed=4 pending=53 cuda1:8511/24564MiB 99% cuda3:1783/24564MiB 15%260817-08:59:58 (UTC+8)
33222026-08-17 08:49:57 done=216/273 running=0 failed=4 pending=53 cuda1:11137/24564MiB 100% cuda3:1783/24564MiB 26%260817-08:49:58 (UTC+8)
33212026-08-17 08:39:57 done=216/273 running=0 failed=4 pending=53 cuda1:11137/24564MiB 100% cuda3:1783/24564MiB 23%260817-08:39:57 (UTC+8)
33202026-08-17 08:29:56 done=216/273 running=0 failed=4 pending=53 cuda1:11137/24564MiB 100% cuda3:1783/24564MiB 23%260817-08:29:57 (UTC+8)
33192026-08-17 08:19:55 done=216/273 running=0 failed=4 pending=53 cuda1:11137/24564MiB 100% cuda3:1783/24564MiB 19%260817-08:19:56 (UTC+8)
33182026-08-17 08:09:54 done=216/273 running=0 failed=4 pending=53 cuda1:11526/24564MiB 100% cuda3:1783/24564MiB 23%260817-08:09:55 (UTC+8)
33172026-08-17 07:59:53 done=216/273 running=0 failed=4 pending=53 cuda1:12248/24564MiB 100% cuda3:1783/24564MiB 23%260817-07:59:54 (UTC+8)
33162026-08-17 07:49:52 done=216/273 running=0 failed=4 pending=53 cuda1:12248/24564MiB 100% cuda3:1783/24564MiB 23%260817-07:49:53 (UTC+8)
33152026-08-17 07:39:51 done=216/273 running=0 failed=4 pending=53 cuda1:1787/24564MiB 97% cuda3:1783/24564MiB 23%260817-07:39:52 (UTC+8)
33142026-08-17 07:29:50 done=216/273 running=0 failed=4 pending=53 cuda1:1787/24564MiB 94% cuda3:1783/24564MiB 23%260817-07:29:51 (UTC+8)
33132026-08-17 07:19:50 done=216/273 running=0 failed=4 pending=53 cuda1:1787/24564MiB 90% cuda3:1783/24564MiB 21%260817-07:19:50 (UTC+8)
33122026-08-17 07:09:49 done=216/273 running=0 failed=4 pending=53 cuda1:1787/24564MiB 94% cuda3:1783/24564MiB 14%260817-07:09:50 (UTC+8)
33112026-08-17 06:59:48 done=216/273 running=0 failed=4 pending=53 cuda1:1787/24564MiB 91% cuda3:1783/24564MiB 17%260817-06:59:49 (UTC+8)
33102026-08-17 06:49:48 done=216/273 running=0 failed=4 pending=53 cuda1:1787/24564MiB 95% cuda3:1783/24564MiB 16%260817-06:49:48 (UTC+8)
33092026-08-17 06:39:47 done=216/273 running=0 failed=4 pending=53 cuda1:1787/24564MiB 91% cuda3:1783/24564MiB 13%260817-06:39:48 (UTC+8)
33082026-08-17 06:29:47 done=216/273 running=0 failed=4 pending=53 cuda1:1787/24564MiB 79% cuda3:1783/24564MiB 14%260817-06:29:47 (UTC+8)
33072026-08-17 06:19:46 done=216/273 running=0 failed=4 pending=53 cuda1:1787/24564MiB 92% cuda3:1783/24564MiB 21%260817-06:19:47 (UTC+8)
33062026-08-17 06:09:46 done=216/273 running=0 failed=4 pending=53 cuda1:1785/24564MiB 97% cuda3:1783/24564MiB 23%260817-06:09:46 (UTC+8)
33052026-08-17 05:59:45 done=216/273 running=0 failed=4 pending=53 cuda1:2152/24564MiB 80% cuda3:1783/24564MiB 19%260817-05:59:46 (UTC+8)
33042026-08-17 05:49:44 done=216/273 running=0 failed=4 pending=53 cuda1:2336/24564MiB 99% cuda3:1783/24564MiB 13%260817-05:49:45 (UTC+8)
33032026-08-17 05:39:43 done=216/273 running=0 failed=4 pending=53 cuda1:2358/24564MiB 99% cuda3:1783/24564MiB 23%260817-05:39:44 (UTC+8)
33022026-08-17 05:29:43 done=216/273 running=0 failed=4 pending=53 cuda1:2456/24564MiB 95% cuda3:1783/24564MiB 14%260817-05:29:43 (UTC+8)
33012026-08-17 05:19:42 done=216/273 running=0 failed=4 pending=53 cuda1:2478/24564MiB 98% cuda3:1783/24564MiB 18%260817-05:19:43 (UTC+8)
33002026-08-17 05:09:41 done=216/273 running=0 failed=4 pending=53 cuda1:2372/24564MiB 90% cuda3:1783/24564MiB 20%260817-05:09:42 (UTC+8)
32992026-08-17 04:59:40 done=216/273 running=0 failed=4 pending=53 cuda1:2372/24564MiB 83% cuda3:1783/24564MiB 9%260817-04:59:41 (UTC+8)
32982026-08-17 04:49:40 done=216/273 running=0 failed=4 pending=53 cuda1:1966/24564MiB 71% cuda3:1783/24564MiB 8%260817-04:49:40 (UTC+8)
32972026-08-17 04:39:39 done=216/273 running=0 failed=4 pending=53 cuda1:1966/24564MiB 67% cuda3:1783/24564MiB 7%260817-04:39:40 (UTC+8)
32962026-08-17 04:29:38 done=216/273 running=0 failed=4 pending=53 cuda1:1455/24564MiB 93% cuda3:1783/24564MiB 0%260817-04:29:39 (UTC+8)
32952026-08-17 04:19:38 done=216/273 running=0 failed=4 pending=53 cuda1:2400/24564MiB 99% cuda3:1783/24564MiB 23%260817-04:19:38 (UTC+8)
32942026-08-17 04:09:37 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 100% cuda3:1783/24564MiB 23%260817-04:09:38 (UTC+8)
32932026-08-17 03:59:36 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 100% cuda3:1783/24564MiB 22%260817-03:59:37 (UTC+8)
32922026-08-17 03:49:36 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 100% cuda3:1783/24564MiB 23%260817-03:49:36 (UTC+8)
32912026-08-17 03:39:35 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 100% cuda3:1783/24564MiB 22%260817-03:39:36 (UTC+8)
32902026-08-17 03:29:35 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 99% cuda3:1783/24564MiB 23%260817-03:29:35 (UTC+8)
32892026-08-17 03:19:34 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 99% cuda3:1783/24564MiB 15%260817-03:19:35 (UTC+8)
32882026-08-17 03:09:34 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 99% cuda3:1783/24564MiB 23%260817-03:09:34 (UTC+8)
32872026-08-17 02:59:33 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 99% cuda3:1783/24564MiB 23%260817-02:59:33 (UTC+8)
32862026-08-17 02:49:32 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 100% cuda3:1783/24564MiB 15%260817-02:49:32 (UTC+8)
32852026-08-17 02:39:31 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 100% cuda3:1783/24564MiB 23%260817-02:39:32 (UTC+8)
32842026-08-17 02:29:30 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 100% cuda3:1783/24564MiB 21%260817-02:29:31 (UTC+8)
32832026-08-17 02:19:30 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 100% cuda3:1783/24564MiB 23%260817-02:19:30 (UTC+8)
32822026-08-17 02:09:29 done=216/273 running=0 failed=4 pending=53 cuda1:9500/24564MiB 99% cuda3:1783/24564MiB 14%260817-02:09:30 (UTC+8)
32812026-08-17 01:59:28 done=216/273 running=0 failed=4 pending=53 cuda1:9502/24564MiB 99% cuda3:1783/24564MiB 24%260817-01:59:29 (UTC+8)
32802026-08-17 01:49:28 done=216/273 running=0 failed=4 pending=53 cuda1:9600/24564MiB 100% cuda3:1783/24564MiB 22%260817-01:49:28 (UTC+8)
32792026-08-17 01:39:27 done=216/273 running=0 failed=4 pending=53 cuda1:9600/24564MiB 100% cuda3:1783/24564MiB 15%260817-01:39:28 (UTC+8)
32782026-08-17 01:29:26 done=216/273 running=0 failed=4 pending=53 cuda1:9600/24564MiB 100% cuda3:1783/24564MiB 21%260817-01:29:27 (UTC+8)
32772026-08-17 01:19:26 done=216/273 running=0 failed=4 pending=53 cuda1:9194/24564MiB 100% cuda3:1783/24564MiB 22%260817-01:19:26 (UTC+8)
32762026-08-17 01:09:25 done=216/273 running=0 failed=4 pending=53 cuda1:9502/24564MiB 99% cuda3:1783/24564MiB 23%260817-01:09:25 (UTC+8)
32752026-08-17 00:59:24 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 100% cuda3:1783/24564MiB 23%260817-00:59:24 (UTC+8)
32742026-08-17 00:49:23 done=216/273 running=0 failed=4 pending=53 cuda1:13548/24564MiB 100% cuda3:1783/24564MiB 22%260817-00:49:24 (UTC+8)
32732026-08-17 00:39:22 done=216/273 running=0 failed=4 pending=53 cuda1:13548/24564MiB 100% cuda3:1783/24564MiB 22%260817-00:39:23 (UTC+8)
32722026-08-17 00:29:21 done=216/273 running=0 failed=4 pending=53 cuda1:13468/24564MiB 100% cuda3:1783/24564MiB 23%260817-00:29:22 (UTC+8)
32712026-08-17 00:19:20 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 100% cuda3:1783/24564MiB 23%260817-00:19:21 (UTC+8)
32702026-08-17 00:09:19 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 100% cuda3:1783/24564MiB 23%260817-00:09:20 (UTC+8)
32692026-08-16 23:59:17 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 99% cuda3:1783/24564MiB 23%260816-23:59:19 (UTC+8)
32682026-08-16 23:49:16 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 100% cuda3:1783/24564MiB 23%260816-23:49:17 (UTC+8)
32672026-08-16 23:39:15 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 100% cuda3:1783/24564MiB 23%260816-23:39:16 (UTC+8)
32662026-08-16 23:29:13 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 100% cuda3:1783/24564MiB 27%260816-23:29:15 (UTC+8)
32652026-08-16 23:19:12 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 100% cuda3:1783/24564MiB 22%260816-23:19:13 (UTC+8)
32642026-08-16 23:09:12 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 92% cuda3:1783/24564MiB 22%260816-23:09:12 (UTC+8)
32632026-08-16 22:59:11 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 96% cuda3:1783/24564MiB 23%260816-22:59:12 (UTC+8)
32622026-08-16 22:49:09 done=216/273 running=0 failed=4 pending=53 cuda1:9602/24564MiB 98% cuda3:1783/24564MiB 23%260816-22:49:11 (UTC+8)
32612026-08-16 22:39:09 done=216/273 running=0 failed=4 pending=53 cuda1:9700/24564MiB 95% cuda3:1783/24564MiB 22%260816-22:39:09 (UTC+8)
32602026-08-16 22:29:08 done=216/273 running=0 failed=4 pending=53 cuda1:9700/24564MiB 99% cuda3:1783/24564MiB 22%260816-22:29:09 (UTC+8)
32592026-08-16 22:19:08 done=216/273 running=0 failed=4 pending=53 cuda1:9128/24564MiB 95% cuda3:1783/24564MiB 23%260816-22:19:08 (UTC+8)
32582026-08-16 22:09:07 done=216/273 running=0 failed=4 pending=53 cuda1:9616/24564MiB 100% cuda3:1783/24564MiB 14%260816-22:09:08 (UTC+8)
32572026-08-16 21:59:07 done=216/273 running=0 failed=4 pending=53 cuda1:9616/24564MiB 100% cuda3:1783/24564MiB 14%260816-21:59:07 (UTC+8)
32562026-08-16 21:49:06 done=216/273 running=0 failed=4 pending=53 cuda1:9616/24564MiB 100% cuda3:1783/24564MiB 13%260816-21:49:06 (UTC+8)
32552026-08-16 21:39:04 done=216/273 running=0 failed=4 pending=53 cuda1:1777/24564MiB 89% cuda3:1783/24564MiB 22%260816-21:39:06 (UTC+8)
32542026-08-16 21:29:03 done=216/273 running=0 failed=4 pending=53 cuda1:9614/24564MiB 100% cuda3:1783/24564MiB 22%260816-21:29:04 (UTC+8)
32532026-08-16 21:19:02 done=216/273 running=0 failed=4 pending=53 cuda1:9614/24564MiB 100% cuda3:1783/24564MiB 21%260816-21:19:03 (UTC+8)
32522026-08-16 21:09:00 done=216/273 running=0 failed=4 pending=53 cuda1:9614/24564MiB 100% cuda3:1783/24564MiB 22%260816-21:09:02 (UTC+8)
32512026-08-16 20:58:59 done=216/273 running=0 failed=4 pending=53 cuda1:9614/24564MiB 100% cuda3:1783/24564MiB 22%260816-20:59:00 (UTC+8)
32502026-08-16 20:48:59 done=216/273 running=0 failed=4 pending=53 cuda1:9614/24564MiB 100% cuda3:1783/24564MiB 21%260816-20:48:59 (UTC+8)
32492026-08-16 20:38:58 done=216/273 running=0 failed=4 pending=53 cuda1:9614/24564MiB 100% cuda3:1783/24564MiB 24%260816-20:38:58 (UTC+8)
32482026-08-16 20:28:57 done=216/273 running=0 failed=4 pending=53 cuda1:9592/24564MiB 100% cuda3:1783/24564MiB 15%260816-20:28:58 (UTC+8)
32472026-08-16 20:18:56 done=216/273 running=0 failed=4 pending=53 cuda1:9592/24564MiB 100% cuda3:1783/24564MiB 15%260816-20:18:57 (UTC+8)
32462026-08-16 20:08:56 done=216/273 running=0 failed=4 pending=53 cuda1:9592/24564MiB 100% cuda3:1783/24564MiB 15%260816-20:08:56 (UTC+8)
32452026-08-16 19:58:55 done=216/273 running=0 failed=4 pending=53 cuda1:9592/24564MiB 100% cuda3:1783/24564MiB 14%260816-19:58:56 (UTC+8)
32442026-08-16 19:48:54 done=216/273 running=0 failed=4 pending=53 cuda1:9592/24564MiB 100% cuda3:1783/24564MiB 4%260816-19:48:55 (UTC+8)
32432026-08-16 19:38:53 done=216/273 running=0 failed=4 pending=53 cuda1:9676/24564MiB 100% cuda3:1783/24564MiB 0%260816-19:38:54 (UTC+8)
32422026-08-16 19:28:53 done=216/273 running=0 failed=4 pending=53 cuda1:9676/24564MiB 100% cuda3:1783/24564MiB 22%260816-19:28:53 (UTC+8)
32412026-08-16 19:18:52 done=216/273 running=0 failed=4 pending=53 cuda1:9676/24564MiB 100% cuda3:1783/24564MiB 24%260816-19:18:52 (UTC+8)
32402026-08-16 19:08:51 done=216/273 running=0 failed=4 pending=53 cuda1:9308/24564MiB 100% cuda3:1783/24564MiB 22%260816-19:08:52 (UTC+8)
32392026-08-16 18:58:50 done=216/273 running=0 failed=4 pending=53 cuda1:11936/24564MiB 100% cuda3:1783/24564MiB 22%260816-18:58:51 (UTC+8)
32382026-08-16 18:48:49 done=216/273 running=0 failed=4 pending=53 cuda1:2346/24564MiB 99% cuda3:1783/24564MiB 23%260816-18:48:50 (UTC+8)
32372026-08-16 18:38:48 done=216/273 running=0 failed=4 pending=53 cuda1:2058/24564MiB 98% cuda3:1783/24564MiB 14%260816-18:38:49 (UTC+8)
32362026-08-16 18:28:47 done=216/273 running=0 failed=4 pending=53 cuda1:1860/24564MiB 98% cuda3:1783/24564MiB 21%260816-18:28:48 (UTC+8)
32352026-08-16 18:18:47 done=216/273 running=0 failed=4 pending=53 cuda1:1982/24564MiB 95% cuda3:1783/24564MiB 20%260816-18:18:47 (UTC+8)
32342026-08-16 18:08:46 done=216/273 running=0 failed=4 pending=53 cuda1:2216/24564MiB 93% cuda3:1783/24564MiB 12%260816-18:08:47 (UTC+8)
32332026-08-16 17:58:45 done=216/273 running=0 failed=4 pending=53 cuda1:2216/24564MiB 93% cuda3:1783/24564MiB 10%260816-17:58:46 (UTC+8)
32322026-08-16 17:48:45 done=216/273 running=0 failed=4 pending=53 cuda1:2216/24564MiB 87% cuda3:1783/24564MiB 10%260816-17:48:45 (UTC+8)
32312026-08-16 17:38:44 done=216/273 running=0 failed=4 pending=53 cuda1:2276/24564MiB 94% cuda3:1783/24564MiB 14%260816-17:38:45 (UTC+8)
32302026-08-16 17:28:44 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 21%260816-17:28:44 (UTC+8)
32292026-08-16 17:18:43 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 23%260816-17:18:44 (UTC+8)
32282026-08-16 17:08:42 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 23%260816-17:08:43 (UTC+8)
32272026-08-16 16:58:42 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 5%260816-16:58:42 (UTC+8)
32262026-08-16 16:48:41 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 100% cuda3:1783/24564MiB 22%260816-16:48:41 (UTC+8)
32252026-08-16 16:38:40 done=216/273 running=0 failed=4 pending=53 cuda1:9536/24564MiB 100% cuda3:1783/24564MiB 23%260816-16:38:41 (UTC+8)
32242026-08-16 16:28:39 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 15%260816-16:28:40 (UTC+8)
32232026-08-16 16:18:39 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 24%260816-16:18:39 (UTC+8)
32222026-08-16 16:08:38 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 24%260816-16:08:39 (UTC+8)
32212026-08-16 15:58:37 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 100% cuda3:1783/24564MiB 23%260816-15:58:38 (UTC+8)
32202026-08-16 15:48:37 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 0%260816-15:48:37 (UTC+8)
32192026-08-16 15:38:36 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 23%260816-15:38:37 (UTC+8)
32182026-08-16 15:28:36 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 23%260816-15:28:36 (UTC+8)
32172026-08-16 15:18:35 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 15%260816-15:18:36 (UTC+8)
32162026-08-16 15:08:34 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 22%260816-15:08:35 (UTC+8)
32152026-08-16 14:58:33 done=216/273 running=0 failed=4 pending=53 cuda1:9442/24564MiB 99% cuda3:1783/24564MiB 22%260816-14:58:34 (UTC+8)
32142026-08-16 14:48:33 done=216/273 running=0 failed=4 pending=53 cuda1:9440/24564MiB 99% cuda3:1783/24564MiB 23%260816-14:48:33 (UTC+8)
32132026-08-16 14:38:31 done=216/273 running=0 failed=4 pending=53 cuda1:9440/24564MiB 99% cuda3:1783/24564MiB 15%260816-14:38:32 (UTC+8)
32122026-08-16 14:28:30 done=216/273 running=0 failed=4 pending=53 cuda1:9440/24564MiB 99% cuda3:1783/24564MiB 23%260816-14:28:31 (UTC+8)
32112026-08-16 14:18:30 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 23%260816-14:18:30 (UTC+8)
32102026-08-16 14:08:29 done=216/273 running=0 failed=4 pending=53 cuda1:13486/24564MiB 100% cuda3:1783/24564MiB 14%260816-14:08:30 (UTC+8)
32092026-08-16 13:58:28 done=216/273 running=0 failed=4 pending=53 cuda1:13580/24564MiB 100% cuda3:1783/24564MiB 20%260816-13:58:29 (UTC+8)
32082026-08-16 13:48:27 done=216/273 running=0 failed=4 pending=53 cuda1:13520/24564MiB 100% cuda3:1783/24564MiB 14%260816-13:48:28 (UTC+8)
32072026-08-16 13:38:26 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 16%260816-13:38:27 (UTC+8)
32062026-08-16 13:28:25 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 99% cuda3:1783/24564MiB 17%260816-13:28:26 (UTC+8)
32052026-08-16 13:18:25 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 0%260816-13:18:25 (UTC+8)
32042026-08-16 13:08:24 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 17%260816-13:08:25 (UTC+8)
32032026-08-16 12:58:24 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 8%260816-12:58:24 (UTC+8)
32022026-08-16 12:48:23 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 14%260816-12:48:24 (UTC+8)
32012026-08-16 12:38:23 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 13%260816-12:38:23 (UTC+8)
32002026-08-16 12:28:22 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 14%260816-12:28:23 (UTC+8)
31992026-08-16 12:18:22 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 17%260816-12:18:22 (UTC+8)
31982026-08-16 12:08:21 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 0%260816-12:08:21 (UTC+8)
31972026-08-16 11:58:20 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 23%260816-11:58:21 (UTC+8)
31962026-08-16 11:48:19 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 24%260816-11:48:20 (UTC+8)
31952026-08-16 11:38:19 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 0%260816-11:38:19 (UTC+8)
31942026-08-16 11:28:18 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 23%260816-11:28:19 (UTC+8)
31932026-08-16 11:18:18 done=216/273 running=0 failed=4 pending=53 cuda1:9540/24564MiB 100% cuda3:1783/24564MiB 24%260816-11:18:18 (UTC+8)
31922026-08-16 11:08:17 done=216/273 running=0 failed=4 pending=53 cuda1:9634/24564MiB 100% cuda3:1783/24564MiB 13%260816-11:08:17 (UTC+8)
31912026-08-16 10:58:16 done=216/273 running=0 failed=4 pending=53 cuda1:9634/24564MiB 100% cuda3:1783/24564MiB 23%260816-10:58:17 (UTC+8)
31902026-08-16 10:48:16 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 23%260816-10:48:16 (UTC+8)
31892026-08-16 10:38:15 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 15%260816-10:38:16 (UTC+8)
31882026-08-16 10:28:14 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 23%260816-10:28:15 (UTC+8)
31872026-08-16 10:18:13 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 22%260816-10:18:14 (UTC+8)
31862026-08-16 10:08:12 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 23%260816-10:08:13 (UTC+8)
31852026-08-16 09:58:11 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 15%260816-09:58:12 (UTC+8)
31842026-08-16 09:48:10 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 23%260816-09:48:11 (UTC+8)
31832026-08-16 09:38:10 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 23%260816-09:38:10 (UTC+8)
31822026-08-16 09:28:09 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 18%260816-09:28:09 (UTC+8)
31812026-08-16 09:18:08 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 0%260816-09:18:09 (UTC+8)
31802026-08-16 09:08:07 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 24%260816-09:08:08 (UTC+8)
31792026-08-16 08:58:06 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 23%260816-08:58:07 (UTC+8)
31782026-08-16 08:48:06 done=216/273 running=0 failed=4 pending=53 cuda1:9546/24564MiB 100% cuda3:1783/24564MiB 15%260816-08:48:06 (UTC+8)
31772026-08-16 08:38:05 done=216/273 running=0 failed=4 pending=53 cuda1:12174/24564MiB 100% cuda3:1783/24564MiB 22%260816-08:38:06 (UTC+8)
31762026-08-16 08:28:05 done=216/273 running=0 failed=4 pending=53 cuda1:12172/24564MiB 100% cuda3:1783/24564MiB 23%260816-08:28:05 (UTC+8)
31752026-08-16 08:18:04 done=216/273 running=0 failed=4 pending=53 cuda1:2382/24564MiB 93% cuda3:1783/24564MiB 23%260816-08:18:05 (UTC+8)
31742026-08-16 08:08:03 done=216/273 running=0 failed=4 pending=53 cuda1:1964/24564MiB 93% cuda3:1783/24564MiB 15%260816-08:08:04 (UTC+8)
31732026-08-16 07:58:02 done=216/273 running=0 failed=4 pending=53 cuda1:1858/24564MiB 17% cuda3:1783/24564MiB 22%260816-07:58:03 (UTC+8)
31722026-08-16 07:48:01 done=216/273 running=0 failed=4 pending=53 cuda1:2090/24564MiB 82% cuda3:1783/24564MiB 13%260816-07:48:02 (UTC+8)
31712026-08-16 07:38:01 done=216/273 running=0 failed=4 pending=53 cuda1:2090/24564MiB 95% cuda3:1783/24564MiB 21%260816-07:38:01 (UTC+8)
31702026-08-16 07:28:00 done=216/273 running=0 failed=4 pending=53 cuda1:2090/24564MiB 97% cuda3:1783/24564MiB 20%260816-07:28:01 (UTC+8)
31692026-08-16 07:17:59 done=216/273 running=0 failed=4 pending=53 cuda1:2452/24564MiB 99% cuda3:1783/24564MiB 0%260816-07:18:00 (UTC+8)
31682026-08-16 07:07:58 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 99% cuda3:1783/24564MiB 23%260816-07:07:59 (UTC+8)
31672026-08-16 06:57:58 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 100% cuda3:1783/24564MiB 23%260816-06:57:58 (UTC+8)
31662026-08-16 06:47:57 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 100% cuda3:1783/24564MiB 14%260816-06:47:58 (UTC+8)
31652026-08-16 06:37:56 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 100% cuda3:1783/24564MiB 23%260816-06:37:57 (UTC+8)
31642026-08-16 06:27:55 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 100% cuda3:1783/24564MiB 23%260816-06:27:56 (UTC+8)
31632026-08-16 06:17:54 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 100% cuda3:1783/24564MiB 15%260816-06:17:55 (UTC+8)
31622026-08-16 06:07:54 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 99% cuda3:1783/24564MiB 23%260816-06:07:54 (UTC+8)
31612026-08-16 05:57:53 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 99% cuda3:1783/24564MiB 17%260816-05:57:54 (UTC+8)
31602026-08-16 05:47:53 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 99% cuda3:1783/24564MiB 23%260816-05:47:53 (UTC+8)
31592026-08-16 05:37:52 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 100% cuda3:1783/24564MiB 15%260816-05:37:52 (UTC+8)
31582026-08-16 05:27:51 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 100% cuda3:1783/24564MiB 22%260816-05:27:52 (UTC+8)
31572026-08-16 05:17:51 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 99% cuda3:1783/24564MiB 22%260816-05:17:51 (UTC+8)
31562026-08-16 05:07:50 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 100% cuda3:1783/24564MiB 14%260816-05:07:50 (UTC+8)
31552026-08-16 04:57:49 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 100% cuda3:1783/24564MiB 23%260816-04:57:50 (UTC+8)
31542026-08-16 04:47:49 done=216/273 running=0 failed=4 pending=53 cuda1:9316/24564MiB 99% cuda3:1783/24564MiB 23%260816-04:47:49 (UTC+8)
31532026-08-16 04:37:48 done=216/273 running=0 failed=4 pending=53 cuda1:9314/24564MiB 99% cuda3:1783/24564MiB 22%260816-04:37:49 (UTC+8)
31522026-08-16 04:27:47 done=216/273 running=0 failed=4 pending=53 cuda1:9314/24564MiB 100% cuda3:1783/24564MiB 14%260816-04:27:48 (UTC+8)
31512026-08-16 04:17:47 done=216/273 running=0 failed=4 pending=53 cuda1:9314/24564MiB 99% cuda3:1783/24564MiB 24%260816-04:17:47 (UTC+8)
31502026-08-16 04:07:46 done=216/273 running=0 failed=4 pending=53 cuda1:9314/24564MiB 99% cuda3:1783/24564MiB 22%260816-04:07:46 (UTC+8)
31492026-08-16 03:57:45 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:1783/24564MiB 23%260816-03:57:46 (UTC+8)
31482026-08-16 03:47:44 done=216/273 running=0 failed=4 pending=53 cuda1:13362/24564MiB 100% cuda3:1783/24564MiB 0%260816-03:47:45 (UTC+8)
31472026-08-16 03:37:44 done=216/273 running=0 failed=4 pending=53 cuda1:13362/24564MiB 100% cuda3:1783/24564MiB 23%260816-03:37:44 (UTC+8)
31462026-08-16 03:27:41 done=216/273 running=0 failed=4 pending=53 cuda1:13320/24564MiB 100% cuda3:1783/24564MiB 18%260816-03:27:44 (UTC+8)
31452026-08-16 03:17:41 done=216/273 running=0 failed=4 pending=53 cuda1:9454/24564MiB 100% cuda3:1783/24564MiB 15%260816-03:17:41 (UTC+8)
31442026-08-16 03:07:40 done=216/273 running=0 failed=4 pending=53 cuda1:9454/24564MiB 100% cuda3:1783/24564MiB 23%260816-03:07:41 (UTC+8)
31432026-08-16 02:57:40 done=216/273 running=0 failed=4 pending=53 cuda1:9454/24564MiB 100% cuda3:3687/24564MiB 97%260816-02:57:40 (UTC+8)
31422026-08-16 02:47:39 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 91%260816-02:47:40 (UTC+8)
31412026-08-16 02:37:38 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 99% cuda3:3687/24564MiB 87%260816-02:37:39 (UTC+8)
31402026-08-16 02:27:38 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 92%260816-02:27:38 (UTC+8)
31392026-08-16 02:17:37 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 94%260816-02:17:38 (UTC+8)
31382026-08-16 02:07:36 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 85%260816-02:07:37 (UTC+8)
31372026-08-16 01:57:35 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 93%260816-01:57:36 (UTC+8)
31362026-08-16 01:47:34 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 91%260816-01:47:35 (UTC+8)
31352026-08-16 01:37:33 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 96% cuda3:3687/24564MiB 94%260816-01:37:34 (UTC+8)
31342026-08-16 01:27:33 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 94% cuda3:3687/24564MiB 96%260816-01:27:33 (UTC+8)
31332026-08-16 01:17:32 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 53%260816-01:17:33 (UTC+8)
31322026-08-16 01:07:31 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 37%260816-01:07:32 (UTC+8)
31312026-08-16 00:57:31 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 97%260816-00:57:31 (UTC+8)
31302026-08-16 00:47:30 done=216/273 running=0 failed=4 pending=53 cuda1:9266/24564MiB 98% cuda3:3687/24564MiB 16%260816-00:47:31 (UTC+8)
31292026-08-16 00:37:29 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 98% cuda3:3687/24564MiB 95%260816-00:37:30 (UTC+8)
31282026-08-16 00:27:29 done=216/273 running=0 failed=4 pending=53 cuda1:9414/24564MiB 100% cuda3:3687/24564MiB 92%260816-00:27:29 (UTC+8)
31272026-08-16 00:17:28 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 93%260816-00:17:28 (UTC+8)
31262026-08-16 00:07:27 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 92%260816-00:07:28 (UTC+8)
31252026-08-15 23:57:27 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 88%260815-23:57:27 (UTC+8)
31242026-08-15 23:47:26 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 88%260815-23:47:27 (UTC+8)
31232026-08-15 23:37:26 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 41%260815-23:37:26 (UTC+8)
31222026-08-15 23:27:25 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 84%260815-23:27:26 (UTC+8)
31212026-08-15 23:17:23 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 94%260815-23:17:25 (UTC+8)
31202026-08-15 23:07:22 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 97%260815-23:07:23 (UTC+8)
31192026-08-15 22:57:22 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 84%260815-22:57:22 (UTC+8)
31182026-08-15 22:47:21 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 95%260815-22:47:22 (UTC+8)
31172026-08-15 22:37:20 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 95%260815-22:37:21 (UTC+8)
31162026-08-15 22:27:20 done=216/273 running=0 failed=4 pending=53 cuda1:9416/24564MiB 100% cuda3:3687/24564MiB 94%260815-22:27:20 (UTC+8)
31152026-08-15 22:17:18 done=216/273 running=0 failed=4 pending=53 cuda1:9208/24564MiB 97% cuda3:3687/24564MiB 93%260815-22:17:19 (UTC+8)
31142026-08-15 22:07:17 done=216/273 running=0 failed=4 pending=53 cuda1:12042/24564MiB 99% cuda3:3687/24564MiB 84%260815-22:07:17 (UTC+8)
31132026-08-15 21:57:16 done=216/273 running=0 failed=4 pending=53 cuda1:2506/24564MiB 96% cuda3:3687/24564MiB 94%260815-21:57:17 (UTC+8)
31122026-08-15 21:47:15 done=216/273 running=0 failed=4 pending=53 cuda1:1888/24564MiB 99% cuda3:3687/24564MiB 93%260815-21:47:16 (UTC+8)
31112026-08-15 21:37:15 done=216/273 running=0 failed=4 pending=53 cuda1:1888/24564MiB 99% cuda3:3687/24564MiB 77%260815-21:37:15 (UTC+8)
31102026-08-15 21:27:14 done=216/273 running=0 failed=4 pending=53 cuda1:1890/24564MiB 85% cuda3:3687/24564MiB 88%260815-21:27:15 (UTC+8)
31092026-08-15 21:17:12 done=216/273 running=0 failed=4 pending=53 cuda1:1956/24564MiB 99% cuda3:3687/24564MiB 95%260815-21:17:14 (UTC+8)
31082026-08-15 21:07:12 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:3687/24564MiB 90%260815-21:07:12 (UTC+8)
31072026-08-15 20:57:11 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:3687/24564MiB 92%260815-20:57:12 (UTC+8)
31062026-08-15 20:47:10 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:3687/24564MiB 92%260815-20:47:11 (UTC+8)
31052026-08-15 20:37:10 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:3687/24564MiB 93%260815-20:37:10 (UTC+8)
31042026-08-15 20:27:09 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 100% cuda3:3687/24564MiB 91%260815-20:27:09 (UTC+8)
31032026-08-15 20:17:08 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:3687/24564MiB 84%260815-20:17:09 (UTC+8)
31022026-08-15 20:07:07 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:3687/24564MiB 97%260815-20:07:08 (UTC+8)
31012026-08-15 19:57:07 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 100% cuda3:3687/24564MiB 82%260815-19:57:07 (UTC+8)
31002026-08-15 19:47:06 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:3687/24564MiB 72%260815-19:47:06 (UTC+8)
30992026-08-15 19:37:05 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:3687/24564MiB 94%260815-19:37:06 (UTC+8)
30982026-08-15 19:27:05 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 100% cuda3:3687/24564MiB 94%260815-19:27:05 (UTC+8)
30972026-08-15 19:17:04 done=216/273 running=0 failed=4 pending=53 cuda1:9116/24564MiB 99% cuda3:3687/24564MiB 96%260815-19:17:05 (UTC+8)
30962026-08-15 19:07:02 done=216/273 running=0 failed=4 pending=53 cuda1:9116/24564MiB 99% cuda3:1783/24564MiB 21%260815-19:07:04 (UTC+8)
30952026-08-15 18:57:01 done=216/273 running=0 failed=4 pending=53 cuda1:9116/24564MiB 99% cuda3:1783/24564MiB 23%260815-18:57:02 (UTC+8)
30942026-08-15 18:47:00 done=216/273 running=0 failed=4 pending=53 cuda1:9116/24564MiB 99% cuda3:1783/24564MiB 13%260815-18:47:01 (UTC+8)
30932026-08-15 18:37:00 done=216/273 running=0 failed=4 pending=53 cuda1:9116/24564MiB 99% cuda3:1783/24564MiB 17%260815-18:37:00 (UTC+8)
30922026-08-15 18:26:59 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 17%260815-18:26:59 (UTC+8)
30912026-08-15 18:16:58 done=216/273 running=0 failed=4 pending=53 cuda1:13184/24564MiB 100% cuda3:1783/24564MiB 22%260815-18:16:59 (UTC+8)
30902026-08-15 18:06:58 done=216/273 running=0 failed=4 pending=53 cuda1:13184/24564MiB 100% cuda3:1783/24564MiB 0%260815-18:06:58 (UTC+8)
30892026-08-15 17:56:57 done=216/273 running=0 failed=4 pending=53 cuda1:13164/24564MiB 100% cuda3:1783/24564MiB 21%260815-17:56:58 (UTC+8)
30882026-08-15 17:46:56 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 23%260815-17:46:57 (UTC+8)
30872026-08-15 17:36:55 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 0%260815-17:36:56 (UTC+8)
30862026-08-15 17:26:55 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 99% cuda3:1783/24564MiB 24%260815-17:26:55 (UTC+8)
30852026-08-15 17:16:54 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 23%260815-17:16:55 (UTC+8)
30842026-08-15 17:06:53 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 15%260815-17:06:54 (UTC+8)
30832026-08-15 16:56:52 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 23%260815-16:56:53 (UTC+8)
30822026-08-15 16:46:52 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 94% cuda3:1783/24564MiB 23%260815-16:46:52 (UTC+8)
30812026-08-15 16:36:51 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 97% cuda3:1783/24564MiB 23%260815-16:36:52 (UTC+8)
30802026-08-15 16:26:51 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 97% cuda3:1783/24564MiB 14%260815-16:26:51 (UTC+8)
30792026-08-15 16:16:50 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 20%260815-16:16:51 (UTC+8)
30782026-08-15 16:06:49 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 99% cuda3:1783/24564MiB 23%260815-16:06:50 (UTC+8)
30772026-08-15 15:56:48 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 24%260815-15:56:49 (UTC+8)
30762026-08-15 15:46:45 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 0%260815-15:46:48 (UTC+8)
30752026-08-15 15:36:44 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 23%260815-15:36:44 (UTC+8)
30742026-08-15 15:26:43 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 22%260815-15:26:44 (UTC+8)
30732026-08-15 15:16:43 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 14%260815-15:16:43 (UTC+8)
30722026-08-15 15:06:42 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 23%260815-15:06:42 (UTC+8)
30712026-08-15 14:56:41 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 23%260815-14:56:42 (UTC+8)
30702026-08-15 14:46:41 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 23%260815-14:46:41 (UTC+8)
30692026-08-15 14:36:40 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 0%260815-14:36:41 (UTC+8)
30682026-08-15 14:26:40 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 24%260815-14:26:40 (UTC+8)
30672026-08-15 14:16:38 done=216/273 running=0 failed=4 pending=53 cuda1:9024/24564MiB 94% cuda3:1783/24564MiB 22%260815-14:16:40 (UTC+8)
30662026-08-15 14:06:37 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 14%260815-14:06:38 (UTC+8)
30652026-08-15 13:56:37 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 23%260815-13:56:37 (UTC+8)
30642026-08-15 13:46:36 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 22%260815-13:46:37 (UTC+8)
30632026-08-15 13:36:35 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 20%260815-13:36:36 (UTC+8)
30622026-08-15 13:26:35 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:1783/24564MiB 0%260815-13:26:35 (UTC+8)
30612026-08-15 13:16:34 done=216/273 running=0 failed=4 pending=53 cuda1:5444/24564MiB 97% cuda3:1783/24564MiB 22%260815-13:16:34 (UTC+8)
30602026-08-15 13:06:33 done=216/273 running=0 failed=4 pending=53 cuda1:1766/24564MiB 96% cuda3:1783/24564MiB 23%260815-13:06:33 (UTC+8)
30592026-08-15 12:56:32 done=216/273 running=0 failed=4 pending=53 cuda1:11842/24564MiB 100% cuda3:1783/24564MiB 0%260815-12:56:33 (UTC+8)
30582026-08-15 12:46:32 done=216/273 running=0 failed=4 pending=53 cuda1:2264/24564MiB 95% cuda3:1783/24564MiB 15%260815-12:46:32 (UTC+8)
30572026-08-15 12:36:31 done=216/273 running=0 failed=4 pending=53 cuda1:2264/24564MiB 95% cuda3:1783/24564MiB 8%260815-12:36:32 (UTC+8)
30562026-08-15 12:26:31 done=216/273 running=0 failed=4 pending=53 cuda1:1379/24564MiB 95% cuda3:1783/24564MiB 15%260815-12:26:31 (UTC+8)
30552026-08-15 12:16:29 done=216/273 running=0 failed=4 pending=53 cuda1:1890/24564MiB 70% cuda3:1783/24564MiB 6%260815-12:16:30 (UTC+8)
30542026-08-15 12:06:29 done=216/273 running=0 failed=4 pending=53 cuda1:1608/24564MiB 21% cuda3:1783/24564MiB 12%260815-12:06:29 (UTC+8)
30532026-08-15 11:56:28 done=216/273 running=0 failed=4 pending=53 cuda1:1892/24564MiB 98% cuda3:1783/24564MiB 10%260815-11:56:29 (UTC+8)
30522026-08-15 11:46:28 done=216/273 running=0 failed=4 pending=53 cuda1:1890/24564MiB 95% cuda3:1783/24564MiB 19%260815-11:46:28 (UTC+8)
30512026-08-15 11:36:27 done=216/273 running=0 failed=4 pending=53 cuda1:1890/24564MiB 94% cuda3:1783/24564MiB 11%260815-11:36:28 (UTC+8)
30502026-08-15 11:26:26 done=216/273 running=0 failed=4 pending=53 cuda1:1932/24564MiB 99% cuda3:1783/24564MiB 23%260815-11:26:27 (UTC+8)
30492026-08-15 11:16:25 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:1783/24564MiB 23%260815-11:16:26 (UTC+8)
30482026-08-15 11:06:25 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 100% cuda3:1783/24564MiB 23%260815-11:06:25 (UTC+8)
30472026-08-15 10:56:24 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:1783/24564MiB 0%260815-10:56:25 (UTC+8)
30462026-08-15 10:46:24 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:1783/24564MiB 20%260815-10:46:24 (UTC+8)
30452026-08-15 10:36:23 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:1783/24564MiB 23%260815-10:36:24 (UTC+8)
30442026-08-15 10:26:22 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:1783/24564MiB 13%260815-10:26:23 (UTC+8)
30432026-08-15 10:06:13 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:1783/24564MiB 22%260815-10:06:14 (UTC+8)
30422026-08-15 09:56:13 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:1783/24564MiB 22%260815-09:56:13 (UTC+8)
30412026-08-15 09:46:12 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:1783/24564MiB 15%260815-09:46:13 (UTC+8)
30402026-08-15 09:36:11 done=216/273 running=0 failed=4 pending=53 cuda1:9182/24564MiB 99% cuda3:1783/24564MiB 23%260815-09:36:12 (UTC+8)
30392026-08-15 09:26:11 done=216/273 running=0 failed=4 pending=53 cuda1:9116/24564MiB 99% cuda3:1783/24564MiB 20%260815-09:26:11 (UTC+8)
30382026-08-15 09:16:10 done=216/273 running=0 failed=4 pending=53 cuda1:9116/24564MiB 100% cuda3:1783/24564MiB 22%260815-09:16:11 (UTC+8)
30372026-08-15 09:06:09 done=216/273 running=0 failed=4 pending=53 cuda1:9116/24564MiB 99% cuda3:1783/24564MiB 0%260815-09:06:10 (UTC+8)
30362026-08-15 08:56:07 done=216/273 running=0 failed=4 pending=53 cuda1:9116/24564MiB 100% cuda3:1783/24564MiB 24%260815-08:56:09 (UTC+8)
30352026-08-15 08:46:07 done=216/273 running=0 failed=4 pending=53 cuda1:9114/24564MiB 100% cuda3:1783/24564MiB 20%260815-08:46:07 (UTC+8)
30342026-08-15 08:36:06 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 15%260815-08:36:07 (UTC+8)
30332026-08-15 08:26:05 done=216/273 running=0 failed=4 pending=53 cuda1:13202/24564MiB 100% cuda3:1783/24564MiB 23%260815-08:26:06 (UTC+8)
30322026-08-15 08:16:04 done=216/273 running=0 failed=4 pending=53 cuda1:13200/24564MiB 100% cuda3:1783/24564MiB 16%260815-08:16:05 (UTC+8)
30312026-08-15 08:06:04 done=216/273 running=0 failed=4 pending=53 cuda1:13200/24564MiB 100% cuda3:1783/24564MiB 0%260815-08:06:04 (UTC+8)
30302026-08-15 07:56:03 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 18%260815-07:56:04 (UTC+8)
30292026-08-15 07:46:03 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:1783/24564MiB 20%260815-07:46:03 (UTC+8)
30282026-08-15 07:36:02 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 86%260815-07:36:03 (UTC+8)
30272026-08-15 07:26:01 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 94%260815-07:26:02 (UTC+8)
30262026-08-15 07:16:00 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 93%260815-07:16:01 (UTC+8)
30252026-08-15 07:06:00 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 95%260815-07:06:00 (UTC+8)
30242026-08-15 06:55:59 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 91%260815-06:56:00 (UTC+8)
30232026-08-15 06:45:58 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 88%260815-06:45:59 (UTC+8)
30222026-08-15 06:35:58 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 94%260815-06:35:58 (UTC+8)
30212026-08-15 06:25:57 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 93% cuda3:3687/24564MiB 84%260815-06:25:58 (UTC+8)
30202026-08-15 06:15:56 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 94%260815-06:15:57 (UTC+8)
30192026-08-15 06:05:56 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 92%260815-06:05:56 (UTC+8)
30182026-08-15 05:55:54 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 95%260815-05:55:56 (UTC+8)
30172026-08-15 05:45:53 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 91%260815-05:45:54 (UTC+8)
30162026-08-15 05:35:52 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 82%260815-05:35:53 (UTC+8)
30152026-08-15 05:25:51 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 95%260815-05:25:52 (UTC+8)
30142026-08-15 05:15:51 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 89%260815-05:15:51 (UTC+8)
30132026-08-15 05:05:50 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 88%260815-05:05:51 (UTC+8)
30122026-08-15 04:55:50 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 92%260815-04:55:50 (UTC+8)
30112026-08-15 04:45:49 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 95%260815-04:45:49 (UTC+8)
30102026-08-15 04:35:48 done=216/273 running=0 failed=4 pending=53 cuda1:9214/24564MiB 100% cuda3:3687/24564MiB 97%260815-04:35:49 (UTC+8)
30092026-08-15 04:25:48 done=216/273 running=0 failed=4 pending=53 cuda1:9030/24564MiB 96% cuda3:3687/24564MiB 84%260815-04:25:48 (UTC+8)
30082026-08-15 04:15:47 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:3687/24564MiB 36%260815-04:15:48 (UTC+8)
30072026-08-15 04:05:47 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:3687/24564MiB 87%260815-04:05:47 (UTC+8)
30062026-08-15 03:55:46 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:3687/24564MiB 92%260815-03:55:46 (UTC+8)
30052026-08-15 03:45:45 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:3687/24564MiB 94%260815-03:45:46 (UTC+8)
30042026-08-15 03:35:45 done=216/273 running=0 failed=4 pending=53 cuda1:9216/24564MiB 100% cuda3:3687/24564MiB 95%260815-03:35:45 (UTC+8)
30032026-08-15 03:25:44 done=216/273 running=0 failed=4 pending=53 cuda1:1766/24564MiB 97% cuda3:3687/24564MiB 95%260815-03:25:44 (UTC+8)
30022026-08-15 03:15:43 done=216/273 running=0 failed=4 pending=53 cuda1:1766/24564MiB 95% cuda3:3687/24564MiB 86%260815-03:15:43 (UTC+8)
30012026-08-15 03:05:42 done=216/273 running=0 failed=4 pending=53 cuda1:11842/24564MiB 100% cuda3:3687/24564MiB 95%260815-03:05:43 (UTC+8)
30002026-08-15 02:55:42 done=216/273 running=0 failed=4 pending=53 cuda1:1930/24564MiB 94% cuda3:3687/24564MiB 84%260815-02:55:42 (UTC+8)
29992026-08-15 02:45:41 done=216/273 running=0 failed=4 pending=53 cuda1:1890/24564MiB 99% cuda3:3687/24564MiB 89%260815-02:45:42 (UTC+8)
29982026-08-15 02:35:40 done=216/273 running=0 failed=4 pending=53 cuda1:1608/24564MiB 85% cuda3:3687/24564MiB 89%260815-02:35:41 (UTC+8)
29972026-08-15 02:25:39 done=216/273 running=0 failed=4 pending=53 cuda1:1892/24564MiB 70% cuda3:3687/24564MiB 78%260815-02:25:40 (UTC+8)
29962026-08-15 02:15:37 done=216/273 running=0 failed=4 pending=53 cuda1:1892/24564MiB 83% cuda3:3687/24564MiB 85%260815-02:15:39 (UTC+8)
29952026-08-15 02:05:36 done=216/273 running=0 failed=4 pending=53 cuda1:2354/24564MiB 99% cuda3:3687/24564MiB 92%260815-02:05:37 (UTC+8)
29942026-08-15 01:55:35 done=216/273 running=0 failed=4 pending=53 cuda1:1381/24564MiB 91% cuda3:3687/24564MiB 72%260815-01:55:36 (UTC+8)
29932026-08-15 01:45:35 done=216/273 running=0 failed=4 pending=53 cuda1:2218/24564MiB 99% cuda3:3687/24564MiB 96%260815-01:45:35 (UTC+8)
29922026-08-15 01:35:34 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:3687/24564MiB 88%260815-01:35:35 (UTC+8)
29912026-08-15 01:25:33 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:3687/24564MiB 96%260815-01:25:34 (UTC+8)
29902026-08-15 01:15:32 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:3687/24564MiB 94%260815-01:15:33 (UTC+8)
29892026-08-15 01:05:31 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:3687/24564MiB 96%260815-01:05:32 (UTC+8)
29882026-08-15 00:55:31 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:3687/24564MiB 94%260815-00:55:31 (UTC+8)
29872026-08-15 00:45:30 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:3687/24564MiB 84%260815-00:45:31 (UTC+8)
29862026-08-15 00:35:29 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 100% cuda3:3687/24564MiB 88%260815-00:35:30 (UTC+8)
29852026-08-15 00:25:28 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:3687/24564MiB 94%260815-00:25:29 (UTC+8)
29842026-08-15 00:15:28 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:3687/24564MiB 50%260815-00:15:28 (UTC+8)
29832026-08-15 00:05:27 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:3687/24564MiB 92%260815-00:05:28 (UTC+8)
29822026-08-14 23:55:26 done=216/273 running=0 failed=4 pending=53 cuda1:9158/24564MiB 99% cuda3:1783/24564MiB 17%260814-23:55:27 (UTC+8)
29812026-08-14 23:45:26 done=216/273 running=0 failed=4 pending=53 cuda1:9090/24564MiB 99% cuda3:1783/24564MiB 15%260814-23:45:26 (UTC+8)
29802026-08-14 23:35:25 done=216/273 running=0 failed=4 pending=53 cuda1:9090/24564MiB 99% cuda3:1783/24564MiB 19%260814-23:35:25 (UTC+8)
29792026-08-14 23:25:24 done=216/273 running=0 failed=4 pending=53 cuda1:9090/24564MiB 99% cuda3:1783/24564MiB 21%260814-23:25:24 (UTC+8)
29782026-08-14 23:15:23 done=216/273 running=0 failed=4 pending=53 cuda1:9090/24564MiB 99% cuda3:1783/24564MiB 15%260814-23:15:24 (UTC+8)
29772026-08-14 23:05:23 done=216/273 running=0 failed=4 pending=53 cuda1:9090/24564MiB 99% cuda3:1783/24564MiB 21%260814-23:05:23 (UTC+8)
29762026-08-14 22:55:22 done=216/273 running=0 failed=4 pending=53 cuda1:9090/24564MiB 99% cuda3:1783/24564MiB 19%260814-22:55:22 (UTC+8)
29752026-08-14 22:45:19 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:1783/24564MiB 17%260814-22:45:22 (UTC+8)
29742026-08-14 22:35:18 done=216/273 running=0 failed=4 pending=53 cuda1:13158/24564MiB 100% cuda3:1783/24564MiB 15%260814-22:35:19 (UTC+8)
29732026-08-14 22:25:18 done=216/273 running=0 failed=4 pending=53 cuda1:13138/24564MiB 100% cuda3:1783/24564MiB 17%260814-22:25:18 (UTC+8)
29722026-08-14 22:15:17 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:1783/24564MiB 18%260814-22:15:18 (UTC+8)
29712026-08-14 22:05:15 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 95%260814-22:05:17 (UTC+8)
29702026-08-14 21:55:15 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 93%260814-21:55:15 (UTC+8)
29692026-08-14 21:45:14 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 93%260814-21:45:15 (UTC+8)
29682026-08-14 21:35:13 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 84%260814-21:35:14 (UTC+8)
29672026-08-14 21:25:13 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 96%260814-21:25:13 (UTC+8)
29662026-08-14 21:15:12 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 87%260814-21:15:13 (UTC+8)
29652026-08-14 21:05:11 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 94%260814-21:05:12 (UTC+8)
29642026-08-14 20:55:11 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 93% cuda3:3687/24564MiB 93%260814-20:55:11 (UTC+8)
29632026-08-14 20:45:10 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 92% cuda3:3687/24564MiB 91%260814-20:45:11 (UTC+8)
29622026-08-14 20:35:09 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 89%260814-20:35:10 (UTC+8)
29612026-08-14 20:25:08 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 89% cuda3:3687/24564MiB 80%260814-20:25:09 (UTC+8)
29602026-08-14 20:15:07 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 97% cuda3:3687/24564MiB 96%260814-20:15:08 (UTC+8)
29592026-08-14 20:05:06 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 88%260814-20:05:07 (UTC+8)
29582026-08-14 19:55:05 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 92%260814-19:55:05 (UTC+8)
29572026-08-14 19:45:04 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 82%260814-19:45:05 (UTC+8)
29562026-08-14 19:35:04 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 94%260814-19:35:04 (UTC+8)
29552026-08-14 19:25:03 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 92%260814-19:25:04 (UTC+8)
29542026-08-14 19:15:01 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 75%260814-19:15:03 (UTC+8)
29532026-08-14 19:05:01 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 96%260814-19:05:01 (UTC+8)
29522026-08-14 18:55:00 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 83%260814-18:55:01 (UTC+8)
29512026-08-14 18:44:59 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 87%260814-18:44:59 (UTC+8)
29502026-08-14 18:34:58 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 95%260814-18:34:59 (UTC+8)
29492026-08-14 18:24:56 done=216/273 running=0 failed=4 pending=53 cuda1:9190/24564MiB 100% cuda3:3687/24564MiB 89%260814-18:24:58 (UTC+8)
29482026-08-14 18:14:56 done=216/273 running=0 failed=4 pending=53 cuda1:9192/24564MiB 100% cuda3:3687/24564MiB 18%260814-18:14:56 (UTC+8)
29472026-08-14 18:04:54 done=216/273 running=0 failed=4 pending=53 cuda1:8978/24564MiB 99% cuda3:3687/24564MiB 43%260814-18:04:55 (UTC+8)
29462026-08-14 17:54:51 done=216/273 running=0 failed=4 pending=53 cuda1:9192/24564MiB 100% cuda3:3687/24564MiB 96%260814-17:54:54 (UTC+8)
29452026-08-14 17:44:51 done=216/273 running=0 failed=4 pending=53 cuda1:9192/24564MiB 100% cuda3:3687/24564MiB 96%260814-17:44:51 (UTC+8)
29442026-08-14 17:34:50 done=216/273 running=0 failed=4 pending=53 cuda1:9192/24564MiB 97% cuda3:3687/24564MiB 95%260814-17:34:51 (UTC+8)
29432026-08-14 17:24:50 done=216/273 running=0 failed=4 pending=53 cuda1:11818/24564MiB 100% cuda3:3687/24564MiB 91%260814-17:24:50 (UTC+8)
29422026-08-14 17:14:49 done=216/273 running=0 failed=4 pending=53 cuda1:11818/24564MiB 100% cuda3:3687/24564MiB 67%260814-17:14:50 (UTC+8)
29412026-08-14 17:04:46 done=216/273 running=0 failed=4 pending=53 cuda1:1355/24564MiB 96% cuda3:3687/24564MiB 96%260814-17:04:49 (UTC+8)
29402026-08-14 16:54:45 done=216/273 running=0 failed=4 pending=53 cuda1:1355/24564MiB 98% cuda3:3687/24564MiB 85%260814-16:54:46 (UTC+8)
29392026-08-14 16:44:44 done=216/273 running=0 failed=4 pending=53 cuda1:1355/24564MiB 95% cuda3:3687/24564MiB 54%260814-16:44:45 (UTC+8)
29382026-08-14 16:34:43 done=216/273 running=0 failed=4 pending=53 cuda1:1355/24564MiB 98% cuda3:3687/24564MiB 92%260814-16:34:44 (UTC+8)
29372026-08-14 16:24:42 done=216/273 running=0 failed=4 pending=53 cuda1:1355/24564MiB 96% cuda3:3687/24564MiB 92%260814-16:24:43 (UTC+8)
29362026-08-14 16:14:39 done=216/273 running=0 failed=4 pending=53 cuda1:1355/24564MiB 94% cuda3:3687/24564MiB 93%260814-16:14:42 (UTC+8)
29352026-08-14 16:04:37 done=216/273 running=0 failed=4 pending=53 cuda1:1355/24564MiB 97% cuda3:3687/24564MiB 91%260814-16:04:39 (UTC+8)
29342026-08-14 15:54:35 done=216/273 running=0 failed=4 pending=53 cuda1:1355/24564MiB 93% cuda3:3687/24564MiB 94%260814-15:54:37 (UTC+8)
29332026-08-14 15:44:32 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 15% cuda3:3687/24564MiB 86%260814-15:44:35 (UTC+8)
29322026-08-14 15:34:31 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 14% cuda3:3687/24564MiB 96%260814-15:34:32 (UTC+8)
29312026-08-14 15:24:30 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 15% cuda3:3687/24564MiB 93%260814-15:24:31 (UTC+8)
29302026-08-14 15:14:29 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 15% cuda3:3687/24564MiB 94%260814-15:14:30 (UTC+8)
29292026-08-14 15:04:28 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 15% cuda3:3687/24564MiB 93%260814-15:04:29 (UTC+8)
29282026-08-14 14:54:26 done=216/273 running=0 failed=4 pending=53 cuda1:571/24564MiB 0% cuda3:3687/24564MiB 85%260814-14:54:28 (UTC+8)
29272026-08-14 14:44:25 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 13% cuda3:1783/24564MiB 17%260814-14:44:26 (UTC+8)
29262026-08-14 14:34:24 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:1783/24564MiB 16%260814-14:34:24 (UTC+8)
29252026-08-14 14:24:23 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 13% cuda3:1783/24564MiB 15%260814-14:24:24 (UTC+8)
29242026-08-14 14:14:22 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:1783/24564MiB 17%260814-14:14:23 (UTC+8)
29232026-08-14 14:04:21 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 13% cuda3:6413/24564MiB 80%260814-14:04:22 (UTC+8)
29222026-08-14 13:44:13 done=216/273 running=0 failed=4 pending=53 cuda1:629/24564MiB 13% cuda3:6413/24564MiB 93%260814-13:44:13 (UTC+8)
29212026-08-14 13:34:12 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6411/24564MiB 88%260814-13:34:13 (UTC+8)
29202026-08-14 13:24:12 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6411/24564MiB 88%260814-13:24:12 (UTC+8)
29192026-08-14 13:14:11 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6411/24564MiB 89%260814-13:14:11 (UTC+8)
29182026-08-14 13:04:10 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6411/24564MiB 89%260814-13:04:10 (UTC+8)
29172026-08-14 12:54:09 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6411/24564MiB 89%260814-12:54:10 (UTC+8)
29162026-08-14 12:44:08 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 13% cuda3:6411/24564MiB 85%260814-12:44:09 (UTC+8)
29152026-08-14 12:34:08 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6413/24564MiB 92%260814-12:34:08 (UTC+8)
29142026-08-14 12:24:07 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6413/24564MiB 93%260814-12:24:08 (UTC+8)
29132026-08-14 12:14:07 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 15% cuda3:6413/24564MiB 88%260814-12:14:07 (UTC+8)
29122026-08-14 12:04:06 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 13% cuda3:6413/24564MiB 88%260814-12:04:07 (UTC+8)
29112026-08-14 11:54:05 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6411/24564MiB 93%260814-11:54:06 (UTC+8)
29102026-08-14 11:44:04 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6411/24564MiB 88%260814-11:44:05 (UTC+8)
29092026-08-14 11:34:03 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6411/24564MiB 89%260814-11:34:04 (UTC+8)
29082026-08-14 11:24:03 done=216/273 running=0 failed=4 pending=53 cuda1:631/24564MiB 14% cuda3:6007/24564MiB 93%260814-11:24:03 (UTC+8)
29072026-08-14 11:14:02 done=216/273 running=0 failed=4 pending=53 cuda1:629/24564MiB 12% cuda3:6007/24564MiB 86%260814-11:14:02 (UTC+8)
29062026-08-14 11:04:01 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:4621/24564MiB 27%260814-11:04:01 (UTC+8)
29052026-08-14 10:54:00 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:8451/24564MiB 98%260814-10:54:01 (UTC+8)
29042026-08-14 10:43:59 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:8447/24564MiB 89%260814-10:44:00 (UTC+8)
29032026-08-14 10:33:59 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:8449/24564MiB 100%260814-10:33:59 (UTC+8)
29022026-08-14 10:23:58 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:8449/24564MiB 48%260814-10:23:58 (UTC+8)
29012026-08-14 10:13:57 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:5365/24564MiB 100%260814-10:13:58 (UTC+8)
29002026-08-14 10:03:56 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:4865/24564MiB 97%260814-10:03:57 (UTC+8)
28992026-08-14 09:53:55 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:4565/24564MiB 95%260814-09:53:56 (UTC+8)
28982026-08-14 09:43:54 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:5405/24564MiB 95%260814-09:43:55 (UTC+8)
28972026-08-14 09:33:54 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:5713/24564MiB 98%260814-09:33:54 (UTC+8)
28962026-08-14 09:23:53 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:5713/24564MiB 15%260814-09:23:54 (UTC+8)
28952026-08-14 09:13:52 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:1249/24564MiB 15%260814-09:13:53 (UTC+8)
28942026-08-14 09:03:52 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:6273/24564MiB 100%260814-09:03:52 (UTC+8)
28932026-08-14 08:53:51 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:6273/24564MiB 96%260814-08:53:52 (UTC+8)
28922026-08-14 08:43:50 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:5733/24564MiB 98%260814-08:43:51 (UTC+8)
28912026-08-14 08:33:49 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:6273/24564MiB 96%260814-08:33:50 (UTC+8)
28902026-08-14 08:23:49 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:6273/24564MiB 100%260814-08:23:49 (UTC+8)
28892026-08-14 08:13:47 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:5453/24564MiB 98%260814-08:13:49 (UTC+8)
28882026-08-14 08:03:46 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:5413/24564MiB 99%260814-08:03:47 (UTC+8)
28872026-08-14 07:53:45 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:6293/24564MiB 100%260814-07:53:46 (UTC+8)
28862026-08-14 07:43:44 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:5453/24564MiB 98%260814-07:43:45 (UTC+8)
28852026-08-14 07:33:44 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:5453/24564MiB 98%260814-07:33:44 (UTC+8)
28842026-08-14 07:23:43 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7616/24564MiB 100%260814-07:23:44 (UTC+8)
28832026-08-14 07:13:42 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:10698/24564MiB 100%260814-07:13:43 (UTC+8)
28822026-08-14 07:03:41 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:2765/24564MiB 98%260814-07:03:42 (UTC+8)
28812026-08-14 06:53:40 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:2765/24564MiB 97%260814-06:53:41 (UTC+8)
28802026-08-14 06:43:40 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7392/24564MiB 100%260814-06:43:40 (UTC+8)
28792026-08-14 06:33:39 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7392/24564MiB 99%260814-06:33:40 (UTC+8)
28782026-08-14 06:23:38 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7392/24564MiB 100%260814-06:23:39 (UTC+8)
28772026-08-14 06:13:37 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7392/24564MiB 100%260814-06:13:38 (UTC+8)
28762026-08-14 06:03:36 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7392/24564MiB 100%260814-06:03:37 (UTC+8)
28752026-08-14 05:53:35 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7392/24564MiB 99%260814-05:53:36 (UTC+8)
28742026-08-14 05:43:34 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7392/24564MiB 100%260814-05:43:35 (UTC+8)
28732026-08-14 05:33:33 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7392/24564MiB 100%260814-05:33:34 (UTC+8)
28722026-08-14 05:23:33 done=216/273 running=0 failed=4 pending=53 cuda1:583/24564MiB 0% cuda3:7392/24564MiB 100%260814-05:23:33 (UTC+8)
28712026-08-14 05:13:32 done=216/273 running=0 failed=4 pending=53 cuda1:683/24564MiB 15% cuda3:7392/24564MiB 100%260814-05:13:33 (UTC+8)
28702026-08-14 05:03:32 done=216/273 running=0 failed=4 pending=53 cuda1:683/24564MiB 14% cuda3:7392/24564MiB 100%260814-05:03:32 (UTC+8)
28692026-08-14 04:53:31 done=216/273 running=0 failed=4 pending=53 cuda1:683/24564MiB 14% cuda3:7392/24564MiB 100%260814-04:53:32 (UTC+8)
28682026-08-14 04:43:30 done=216/273 running=0 failed=4 pending=53 cuda1:701/24564MiB 15% cuda3:7392/24564MiB 99%260814-04:43:31 (UTC+8)
28672026-08-14 04:33:29 done=216/273 running=0 failed=4 pending=53 cuda1:701/24564MiB 14% cuda3:7392/24564MiB 100%260814-04:33:30 (UTC+8)
28662026-08-14 04:23:29 done=216/273 running=0 failed=4 pending=53 cuda1:701/24564MiB 15% cuda3:7392/24564MiB 100%260814-04:23:29 (UTC+8)
28652026-08-14 04:13:28 done=216/273 running=0 failed=4 pending=53 cuda1:701/24564MiB 14% cuda3:7392/24564MiB 100%260814-04:13:29 (UTC+8)
28642026-08-14 04:03:28 done=216/273 running=0 failed=4 pending=53 cuda1:701/24564MiB 14% cuda3:7392/24564MiB 100%260814-04:03:28 (UTC+8)
28632026-08-14 03:53:27 done=216/273 running=0 failed=4 pending=53 cuda1:699/24564MiB 14% cuda3:8236/24564MiB 100%260814-03:53:27 (UTC+8)
28622026-08-14 03:43:26 done=216/273 running=0 failed=4 pending=53 cuda1:697/24564MiB 15% cuda3:11220/24564MiB 100%260814-03:43:27 (UTC+8)
28612026-08-14 03:33:26 done=216/273 running=0 failed=4 pending=53 cuda1:697/24564MiB 15% cuda3:11220/24564MiB 100%260814-03:33:26 (UTC+8)
28602026-08-14 03:23:25 done=216/273 running=0 failed=4 pending=53 cuda1:695/24564MiB 14% cuda3:3362/24564MiB 100%260814-03:23:26 (UTC+8)
28592026-08-14 03:13:24 done=216/273 running=0 failed=4 pending=53 cuda1:695/24564MiB 14% cuda3:8176/24564MiB 100%260814-03:13:25 (UTC+8)
28582026-08-14 03:03:23 done=216/273 running=0 failed=4 pending=53 cuda1:695/24564MiB 15% cuda3:7596/24564MiB 100%260814-03:03:24 (UTC+8)
28572026-08-14 02:53:22 done=216/273 running=0 failed=4 pending=53 cuda1:565/24564MiB 0% cuda3:7336/24564MiB 100%260814-02:53:23 (UTC+8)
28562026-08-14 02:43:21 done=216/273 running=0 failed=4 pending=53 cuda1:689/24564MiB 14% cuda3:8196/24564MiB 100%260814-02:43:22 (UTC+8)
28552026-08-14 02:33:21 done=216/273 running=0 failed=4 pending=53 cuda1:689/24564MiB 15% cuda3:8196/24564MiB 100%260814-02:33:21 (UTC+8)
28542026-08-14 02:23:20 done=216/273 running=0 failed=4 pending=53 cuda1:689/24564MiB 14% cuda3:8196/24564MiB 98%260814-02:23:21 (UTC+8)
28532026-08-14 02:13:19 done=216/273 running=0 failed=4 pending=53 cuda1:689/24564MiB 14% cuda3:7636/24564MiB 100%260814-02:13:20 (UTC+8)
28522026-08-14 02:03:19 done=216/273 running=0 failed=4 pending=53 cuda1:687/24564MiB 13% cuda3:7356/24564MiB 98%260814-02:03:19 (UTC+8)
28512026-08-14 01:53:18 done=216/273 running=0 failed=4 pending=53 cuda1:659/24564MiB 14% cuda3:7356/24564MiB 100%260814-01:53:19 (UTC+8)
28502026-08-14 01:43:17 done=216/273 running=0 failed=4 pending=53 cuda1:659/24564MiB 14% cuda3:7316/24564MiB 100%260814-01:43:18 (UTC+8)
28492026-08-14 01:33:17 done=216/273 running=0 failed=4 pending=53 cuda1:659/24564MiB 14% cuda3:8196/24564MiB 100%260814-01:33:17 (UTC+8)
28482026-08-14 01:23:16 done=216/273 running=0 failed=4 pending=53 cuda1:657/24564MiB 14% cuda3:8196/24564MiB 100%260814-01:23:16 (UTC+8)
28472026-08-14 01:13:15 done=216/273 running=0 failed=4 pending=53 cuda1:685/24564MiB 15% cuda3:7636/24564MiB 100%260814-01:13:16 (UTC+8)
28462026-08-14 01:03:15 done=216/273 running=0 failed=4 pending=53 cuda1:685/24564MiB 14% cuda3:8196/24564MiB 100%260814-01:03:15 (UTC+8)
28452026-08-14 00:53:14 done=216/273 running=0 failed=4 pending=53 cuda1:685/24564MiB 14% cuda3:8196/24564MiB 100%260814-00:53:15 (UTC+8)
28442026-08-14 00:43:12 done=216/273 running=0 failed=4 pending=53 cuda1:677/24564MiB 14% cuda3:7636/24564MiB 100%260814-00:43:14 (UTC+8)
28432026-08-14 00:33:12 done=216/273 running=0 failed=4 pending=53 cuda1:677/24564MiB 14% cuda3:7356/24564MiB 100%260814-00:33:12 (UTC+8)
28422026-08-14 00:23:11 done=216/273 running=0 failed=4 pending=53 cuda1:677/24564MiB 14% cuda3:7316/24564MiB 100%260814-00:23:11 (UTC+8)
28412026-08-14 00:13:10 done=216/273 running=0 failed=4 pending=53 cuda1:677/24564MiB 14% cuda3:8196/24564MiB 100%260814-00:13:11 (UTC+8)
28402026-08-14 00:03:09 done=216/273 running=0 failed=4 pending=53 cuda1:643/24564MiB 14% cuda3:7356/24564MiB 100%260814-00:03:10 (UTC+8)
28392026-08-13 23:53:09 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 15% cuda3:7316/24564MiB 98%260813-23:53:09 (UTC+8)
28382026-08-13 23:43:08 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 15% cuda3:7636/24564MiB 100%260813-23:43:09 (UTC+8)
28372026-08-13 23:33:07 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 15% cuda3:7316/24564MiB 100%260813-23:33:08 (UTC+8)
28362026-08-13 23:23:07 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 15% cuda3:10698/24564MiB 100%260813-23:23:07 (UTC+8)
28352026-08-13 23:13:06 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 15% cuda3:8236/24564MiB 100%260813-23:13:06 (UTC+8)
28342026-08-13 23:03:05 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 15% cuda3:11220/24564MiB 100%260813-23:03:06 (UTC+8)
28332026-08-13 22:53:05 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 14% cuda3:11200/24564MiB 100%260813-22:53:05 (UTC+8)
28322026-08-13 22:43:04 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 15% cuda3:2765/24564MiB 98%260813-22:43:05 (UTC+8)
28312026-08-13 22:33:03 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 15% cuda3:2765/24564MiB 97%260813-22:33:04 (UTC+8)
28302026-08-13 22:22:59 done=216/273 running=0 failed=4 pending=53 cuda1:869/24564MiB 15% cuda3:2449/24564MiB 48%260813-22:23:03 (UTC+8)
28292026-08-13 22:12:55 done=216/273 running=0 failed=4 pending=53 cuda1:551/24564MiB 0% cuda3:633/24564MiB 15%260813-22:12:58 (UTC+8)
28282026-08-13 22:02:53 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 15% cuda3:631/24564MiB 14%260813-22:02:55 (UTC+8)
28272026-08-13 21:52:52 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 15% cuda3:2535/24564MiB 97%260813-21:52:53 (UTC+8)
28262026-08-13 21:42:51 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 15% cuda3:2535/24564MiB 96%260813-21:42:52 (UTC+8)
28252026-08-13 21:32:49 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 15% cuda3:2535/24564MiB 96%260813-21:32:51 (UTC+8)
28242026-08-13 21:22:48 done=216/273 running=0 failed=4 pending=53 cuda1:861/24564MiB 15% cuda3:2535/24564MiB 94%260813-21:22:49 (UTC+8)
28232026-08-13 21:12:47 done=216/273 running=0 failed=4 pending=53 cuda1:8699/24564MiB 14% cuda3:10370/24564MiB 100%260813-21:12:48 (UTC+8)
28222026-08-13 21:02:47 done=216/273 running=0 failed=4 pending=53 cuda1:8699/24564MiB 98% cuda3:10370/24564MiB 100%260813-21:02:47 (UTC+8)
28212026-08-13 20:52:46 done=216/273 running=0 failed=4 pending=53 cuda1:8699/24564MiB 15% cuda3:10370/24564MiB 100%260813-20:52:47 (UTC+8)
28202026-08-13 20:42:45 done=216/273 running=0 failed=4 pending=53 cuda1:8699/24564MiB 97% cuda3:10370/24564MiB 100%260813-20:42:46 (UTC+8)
28192026-08-13 20:32:45 done=216/273 running=0 failed=4 pending=53 cuda1:8699/24564MiB 15% cuda3:10370/24564MiB 100%260813-20:32:45 (UTC+8)
28182026-08-13 20:22:44 done=216/273 running=0 failed=4 pending=53 cuda1:8699/24564MiB 80% cuda3:10370/24564MiB 100%260813-20:22:45 (UTC+8)
28172026-08-13 20:12:44 done=216/273 running=0 failed=4 pending=53 cuda1:8699/24564MiB 14% cuda3:10370/24564MiB 100%260813-20:12:44 (UTC+8)
28162026-08-13 20:02:43 done=216/273 running=0 failed=4 pending=53 cuda1:8475/24564MiB 15% cuda3:10370/24564MiB 100%260813-20:02:44 (UTC+8)
28152026-08-13 19:52:43 done=216/273 running=0 failed=4 pending=53 cuda1:8475/24564MiB 14% cuda3:10370/24564MiB 100%260813-19:52:43 (UTC+8)
28142026-08-13 19:42:42 done=216/273 running=0 failed=4 pending=53 cuda1:8475/24564MiB 14% cuda3:10370/24564MiB 100%260813-19:42:43 (UTC+8)
28132026-08-13 19:32:42 done=216/273 running=0 failed=4 pending=53 cuda1:8475/24564MiB 97% cuda3:10370/24564MiB 100%260813-19:32:42 (UTC+8)
28122026-08-13 19:22:41 done=216/273 running=0 failed=4 pending=53 cuda1:8475/24564MiB 14% cuda3:10370/24564MiB 100%260813-19:22:41 (UTC+8)
28112026-08-13 19:12:40 done=216/273 running=0 failed=4 pending=53 cuda1:8475/24564MiB 99% cuda3:10370/24564MiB 100%260813-19:12:41 (UTC+8)
28102026-08-13 19:02:39 done=216/273 running=0 failed=4 pending=53 cuda1:8475/24564MiB 99% cuda3:10370/24564MiB 100%260813-19:02:40 (UTC+8)
28092026-08-13 18:52:39 done=216/273 running=0 failed=4 pending=53 cuda1:8475/24564MiB 14% cuda3:10370/24564MiB 100%260813-18:52:39 (UTC+8)
28082026-08-13 18:42:38 done=216/273 running=0 failed=4 pending=53 cuda1:8475/24564MiB 13% cuda3:10370/24564MiB 100%260813-18:42:38 (UTC+8)
28072026-08-13 18:32:37 done=216/273 running=0 failed=4 pending=53 cuda1:8473/24564MiB 13% cuda3:10370/24564MiB 100%260813-18:32:37 (UTC+8)
28062026-08-13 18:22:35 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-18:22:37 (UTC+8)
28052026-08-13 18:12:35 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 97% cuda3:10370/24564MiB 100%260813-18:12:35 (UTC+8)
28042026-08-13 18:02:34 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 99%260813-18:02:35 (UTC+8)
28032026-08-13 17:52:32 done=216/273 running=0 failed=4 pending=53 cuda1:8858/24564MiB 14% cuda3:10370/24564MiB 100%260813-17:52:34 (UTC+8)
28022026-08-13 17:42:31 done=216/273 running=0 failed=4 pending=53 cuda1:8858/24564MiB 100% cuda3:10370/24564MiB 100%260813-17:42:32 (UTC+8)
28012026-08-13 17:32:28 done=216/273 running=0 failed=4 pending=53 cuda1:8856/24564MiB 100% cuda3:10370/24564MiB 100%260813-17:32:31 (UTC+8)
28002026-08-13 17:22:28 done=216/273 running=0 failed=4 pending=53 cuda1:8856/24564MiB 97% cuda3:10370/24564MiB 100%260813-17:22:28 (UTC+8)
27992026-08-13 17:12:27 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 97% cuda3:10370/24564MiB 100%260813-17:12:28 (UTC+8)
27982026-08-13 17:02:26 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 96% cuda3:10370/24564MiB 100%260813-17:02:27 (UTC+8)
27972026-08-13 16:52:26 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-16:52:26 (UTC+8)
27962026-08-13 16:42:24 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-16:42:25 (UTC+8)
27952026-08-13 16:32:23 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 97% cuda3:10370/24564MiB 100%260813-16:32:24 (UTC+8)
27942026-08-13 16:22:23 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-16:22:23 (UTC+8)
27932026-08-13 16:12:22 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-16:12:23 (UTC+8)
27922026-08-13 16:02:21 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-16:02:22 (UTC+8)
27912026-08-13 15:52:20 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-15:52:21 (UTC+8)
27902026-08-13 15:42:20 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-15:42:20 (UTC+8)
27892026-08-13 15:32:19 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-15:32:19 (UTC+8)
27882026-08-13 15:22:18 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 97% cuda3:10370/24564MiB 100%260813-15:22:19 (UTC+8)
27872026-08-13 15:12:18 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 97% cuda3:10370/24564MiB 100%260813-15:12:18 (UTC+8)
27862026-08-13 15:02:17 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 99% cuda3:10370/24564MiB 100%260813-15:02:18 (UTC+8)
27852026-08-13 14:52:16 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 14% cuda3:10370/24564MiB 99%260813-14:52:17 (UTC+8)
27842026-08-13 14:42:15 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-14:42:16 (UTC+8)
27832026-08-13 14:32:13 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 97% cuda3:10370/24564MiB 100%260813-14:32:14 (UTC+8)
27822026-08-13 14:22:13 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-14:22:13 (UTC+8)
27812026-08-13 14:12:12 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-14:12:13 (UTC+8)
27802026-08-13 14:02:11 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 97% cuda3:10370/24564MiB 100%260813-14:02:12 (UTC+8)
27792026-08-13 13:52:09 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-13:52:11 (UTC+8)
27782026-08-13 13:42:09 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 97% cuda3:10370/24564MiB 100%260813-13:42:09 (UTC+8)
27772026-08-13 13:32:08 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 95% cuda3:10370/24564MiB 100%260813-13:32:09 (UTC+8)
27762026-08-13 13:22:08 done=216/273 running=0 failed=4 pending=53 cuda1:8347/24564MiB 35% cuda3:10370/24564MiB 98%260813-13:22:08 (UTC+8)
27752026-08-13 13:12:06 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-13:12:08 (UTC+8)
27742026-08-13 13:02:05 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-13:02:06 (UTC+8)
27732026-08-13 12:52:05 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-12:52:05 (UTC+8)
27722026-08-13 12:42:04 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 100% cuda3:10370/24564MiB 100%260813-12:42:05 (UTC+8)
27712026-08-13 12:32:03 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-12:32:04 (UTC+8)
27702026-08-13 12:22:03 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-12:22:03 (UTC+8)
27692026-08-13 12:12:01 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:10370/24564MiB 100%260813-12:12:03 (UTC+8)
27682026-08-13 12:02:00 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:12996/24564MiB 100%260813-12:02:01 (UTC+8)
27672026-08-13 11:51:59 done=216/273 running=0 failed=4 pending=53 cuda1:8469/24564MiB 98% cuda3:12996/24564MiB 100%260813-11:52:00 (UTC+8)
27662026-08-13 11:41:59 done=216/273 running=0 failed=4 pending=53 cuda1:8467/24564MiB 98% cuda3:12996/24564MiB 100%260813-11:41:59 (UTC+8)
27652026-08-13 11:31:58 done=216/273 running=0 failed=4 pending=53 cuda1:8467/24564MiB 99% cuda3:11093/24564MiB 100%260813-11:31:59 (UTC+8)
27642026-08-13 11:21:57 done=216/273 running=0 failed=4 pending=53 cuda1:8467/24564MiB 52% cuda3:11093/24564MiB 100%260813-11:21:58 (UTC+8)
27632026-08-13 11:11:56 done=216/273 running=0 failed=4 pending=53 cuda1:8467/24564MiB 100% cuda3:11093/24564MiB 97%260813-11:11:57 (UTC+8)
27622026-08-13 11:01:56 done=216/273 running=0 failed=4 pending=53 cuda1:8467/24564MiB 100% cuda3:11093/24564MiB 100%260813-11:01:56 (UTC+8)
27612026-08-13 10:51:55 done=216/273 running=0 failed=4 pending=53 cuda1:8467/24564MiB 98% cuda3:11093/24564MiB 100%260813-10:51:56 (UTC+8)
27602026-08-13 10:41:54 done=216/273 running=0 failed=4 pending=53 cuda1:8239/24564MiB 100% cuda3:10865/24564MiB 99%260813-10:41:55 (UTC+8)
27592026-08-13 10:31:54 done=216/273 running=0 failed=4 pending=53 cuda1:7683/24564MiB 22% cuda3:10477/24564MiB 100%260813-10:31:54 (UTC+8)
27582026-08-13 10:21:53 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:10477/24564MiB 100%260813-10:21:54 (UTC+8)
27572026-08-13 10:11:53 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:10477/24564MiB 100%260813-10:11:53 (UTC+8)
27562026-08-13 10:01:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:5485/24564MiB 100%260813-10:01:52 (UTC+8)
27552026-08-13 09:51:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 49% cuda3:5465/24564MiB 100%260813-09:51:52 (UTC+8)
27542026-08-13 09:41:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5465/24564MiB 100%260813-09:41:51 (UTC+8)
27532026-08-13 09:31:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5425/24564MiB 100%260813-09:31:50 (UTC+8)
27522026-08-13 09:21:49 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 1% cuda3:8469/24564MiB 99%260813-09:21:50 (UTC+8)
27512026-08-13 09:11:48 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:8469/24564MiB 99%260813-09:11:49 (UTC+8)
27502026-08-13 09:01:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:8469/24564MiB 96%260813-09:01:48 (UTC+8)
27492026-08-13 08:51:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:8469/24564MiB 100%260813-08:51:47 (UTC+8)
27482026-08-13 08:41:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:8469/24564MiB 99%260813-08:41:46 (UTC+8)
27472026-08-13 08:31:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:10373/24564MiB 100%260813-08:31:46 (UTC+8)
27462026-08-13 08:21:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:10373/24564MiB 100%260813-08:21:45 (UTC+8)
27452026-08-13 08:11:43 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:10373/24564MiB 100%260813-08:11:44 (UTC+8)
27442026-08-13 08:01:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:10353/24564MiB 100%260813-08:01:43 (UTC+8)
27432026-08-13 07:51:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:10353/24564MiB 100%260813-07:51:42 (UTC+8)
27422026-08-13 07:41:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:10353/24564MiB 100%260813-07:41:40 (UTC+8)
27412026-08-13 07:31:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:10293/24564MiB 100%260813-07:31:39 (UTC+8)
27402026-08-13 07:21:38 done=216/273 running=0 failed=4 pending=53 cuda1:1631/24564MiB 92% cuda3:10293/24564MiB 100%260813-07:21:39 (UTC+8)
27392026-08-13 07:11:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:2305/24564MiB 45%260813-07:11:38 (UTC+8)
27382026-08-13 07:01:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7349/24564MiB 100%260813-07:01:38 (UTC+8)
27372026-08-13 06:51:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7349/24564MiB 98%260813-06:51:37 (UTC+8)
27362026-08-13 06:41:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7349/24564MiB 92%260813-06:41:36 (UTC+8)
27352026-08-13 06:31:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7349/24564MiB 100%260813-06:31:36 (UTC+8)
27342026-08-13 06:21:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7349/24564MiB 100%260813-06:21:35 (UTC+8)
27332026-08-13 06:11:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7349/24564MiB 77%260813-06:11:34 (UTC+8)
27322026-08-13 06:01:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 57% cuda3:7349/24564MiB 100%260813-06:01:34 (UTC+8)
27312026-08-13 05:51:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6789/24564MiB 99%260813-05:51:33 (UTC+8)
27302026-08-13 05:41:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6789/24564MiB 97%260813-05:41:32 (UTC+8)
27292026-08-13 05:31:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6789/24564MiB 99%260813-05:31:32 (UTC+8)
27282026-08-13 05:21:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6789/24564MiB 97%260813-05:21:31 (UTC+8)
27272026-08-13 05:11:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6769/24564MiB 97%260813-05:11:30 (UTC+8)
27262026-08-13 05:01:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:6509/24564MiB 99%260813-05:01:30 (UTC+8)
27252026-08-13 04:51:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:6529/24564MiB 98%260813-04:51:29 (UTC+8)
27242026-08-13 04:41:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:4367/24564MiB 47%260813-04:41:29 (UTC+8)
27232026-08-13 04:31:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:6469/24564MiB 99%260813-04:31:27 (UTC+8)
27222026-08-13 04:21:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6469/24564MiB 99%260813-04:21:27 (UTC+8)
27212026-08-13 04:11:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 99%260813-04:11:26 (UTC+8)
27202026-08-13 04:01:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7369/24564MiB 46%260813-04:01:25 (UTC+8)
27192026-08-13 03:51:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 100%260813-03:51:25 (UTC+8)
27182026-08-13 03:41:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 49%260813-03:41:24 (UTC+8)
27172026-08-13 03:31:23 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7369/24564MiB 81%260813-03:31:24 (UTC+8)
27162026-08-13 03:21:23 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 100%260813-03:21:23 (UTC+8)
27152026-08-13 03:11:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7349/24564MiB 45%260813-03:11:22 (UTC+8)
27142026-08-13 03:01:21 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7369/24564MiB 96%260813-03:01:22 (UTC+8)
27132026-08-13 02:51:20 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 61%260813-02:51:21 (UTC+8)
27122026-08-13 02:41:20 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7369/24564MiB 46%260813-02:41:20 (UTC+8)
27112026-08-13 02:31:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 100%260813-02:31:20 (UTC+8)
27102026-08-13 02:21:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7369/24564MiB 100%260813-02:21:19 (UTC+8)
27092026-08-13 02:11:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 46%260813-02:11:18 (UTC+8)
27082026-08-13 02:01:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7349/24564MiB 45%260813-02:01:18 (UTC+8)
27072026-08-13 01:51:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:2305/24564MiB 61%260813-01:51:17 (UTC+8)
27062026-08-13 01:41:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6789/24564MiB 99%260813-01:41:17 (UTC+8)
27052026-08-13 01:31:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:6789/24564MiB 100%260813-01:31:16 (UTC+8)
27042026-08-13 01:21:15 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6789/24564MiB 46%260813-01:21:16 (UTC+8)
27032026-08-13 01:11:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6789/24564MiB 47%260813-01:11:15 (UTC+8)
27022026-08-13 01:01:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6789/24564MiB 97%260813-01:01:14 (UTC+8)
27012026-08-13 00:51:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6789/24564MiB 97%260813-00:51:14 (UTC+8)
27002026-08-13 00:41:12 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6789/24564MiB 46%260813-00:41:13 (UTC+8)
26992026-08-13 00:31:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6789/24564MiB 45%260813-00:31:12 (UTC+8)
26982026-08-13 00:21:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6789/24564MiB 48%260813-00:21:11 (UTC+8)
26972026-08-13 00:11:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6789/24564MiB 50%260813-00:11:11 (UTC+8)
26962026-08-13 00:01:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6789/24564MiB 79%260813-00:01:10 (UTC+8)
26952026-08-12 23:51:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6529/24564MiB 48%260812-23:51:10 (UTC+8)
26942026-08-12 23:41:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6529/24564MiB 100%260812-23:41:09 (UTC+8)
26932026-08-12 23:31:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 0%260812-23:31:08 (UTC+8)
26922026-08-12 23:21:07 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 0%260812-23:21:07 (UTC+8)
26912026-08-12 23:11:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 97%260812-23:11:07 (UTC+8)
26902026-08-12 23:01:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:4625/24564MiB 0%260812-23:01:06 (UTC+8)
26892026-08-12 22:51:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 0%260812-22:51:06 (UTC+8)
26882026-08-12 22:41:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 0%260812-22:41:05 (UTC+8)
26872026-08-12 22:31:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4625/24564MiB 0%260812-22:31:04 (UTC+8)
26862026-08-12 22:21:02 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4565/24564MiB 0%260812-22:21:04 (UTC+8)
26852026-08-12 22:11:01 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4565/24564MiB 94%260812-22:11:02 (UTC+8)
26842026-08-12 22:01:01 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4565/24564MiB 97%260812-22:01:01 (UTC+8)
26832026-08-12 21:51:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4565/24564MiB 0%260812-21:51:01 (UTC+8)
26822026-08-12 21:40:59 done=216/273 running=0 failed=4 pending=53 cuda1:7835/24564MiB 0% cuda3:6469/24564MiB 46%260812-21:41:00 (UTC+8)
26812026-08-12 21:30:59 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6469/24564MiB 55%260812-21:30:59 (UTC+8)
26802026-08-12 21:20:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6469/24564MiB 46%260812-21:20:59 (UTC+8)
26792026-08-12 21:10:57 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6469/24564MiB 45%260812-21:10:58 (UTC+8)
26782026-08-12 21:00:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6469/24564MiB 64%260812-21:00:57 (UTC+8)
26772026-08-12 20:50:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6469/24564MiB 45%260812-20:50:56 (UTC+8)
26762026-08-12 20:40:55 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 98%260812-20:40:56 (UTC+8)
26752026-08-12 20:30:55 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 96%260812-20:30:55 (UTC+8)
26742026-08-12 20:20:54 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7369/24564MiB 98%260812-20:20:55 (UTC+8)
26732026-08-12 20:10:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6313/24564MiB 54%260812-20:10:54 (UTC+8)
26722026-08-12 20:00:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7369/24564MiB 100%260812-20:00:52 (UTC+8)
26712026-08-12 19:50:49 done=216/273 running=0 failed=4 pending=53 cuda1:7835/24564MiB 0% cuda3:7369/24564MiB 96%260812-19:50:51 (UTC+8)
26702026-08-12 19:40:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7369/24564MiB 100%260812-19:40:48 (UTC+8)
26692026-08-12 19:30:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6809/24564MiB 97%260812-19:30:46 (UTC+8)
26682026-08-12 19:20:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6789/24564MiB 100%260812-19:20:45 (UTC+8)
26672026-08-12 19:10:43 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6789/24564MiB 100%260812-19:10:44 (UTC+8)
26662026-08-12 19:00:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6789/24564MiB 100%260812-19:00:43 (UTC+8)
26652026-08-12 18:50:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6789/24564MiB 98%260812-18:50:42 (UTC+8)
26642026-08-12 18:40:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6469/24564MiB 99%260812-18:40:42 (UTC+8)
26632026-08-12 18:30:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6469/24564MiB 100%260812-18:30:41 (UTC+8)
26622026-08-12 18:20:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7369/24564MiB 100%260812-18:20:41 (UTC+8)
26612026-08-12 18:10:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7369/24564MiB 96%260812-18:10:40 (UTC+8)
26602026-08-12 18:00:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:7349/24564MiB 100%260812-18:00:39 (UTC+8)
26592026-08-12 17:50:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 100%260812-17:50:38 (UTC+8)
26582026-08-12 17:40:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7369/24564MiB 100%260812-17:40:38 (UTC+8)
26572026-08-12 17:30:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7369/24564MiB 100%260812-17:30:37 (UTC+8)
26562026-08-12 17:20:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 96%260812-17:20:37 (UTC+8)
26552026-08-12 17:10:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7349/24564MiB 100%260812-17:10:36 (UTC+8)
26542026-08-12 17:00:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6789/24564MiB 99%260812-17:00:35 (UTC+8)
26532026-08-12 16:50:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6789/24564MiB 97%260812-16:50:34 (UTC+8)
26522026-08-12 16:40:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6789/24564MiB 100%260812-16:40:33 (UTC+8)
26512026-08-12 16:30:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6789/24564MiB 100%260812-16:30:32 (UTC+8)
26502026-08-12 16:20:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6789/24564MiB 100%260812-16:20:31 (UTC+8)
26492026-08-12 16:10:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6529/24564MiB 100%260812-16:10:31 (UTC+8)
26482026-08-12 16:00:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6529/24564MiB 99%260812-16:00:30 (UTC+8)
26472026-08-12 15:50:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6529/24564MiB 98%260812-15:50:25 (UTC+8)
26462026-08-12 15:40:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6469/24564MiB 98%260812-15:40:24 (UTC+8)
26452026-08-12 15:30:23 done=216/273 running=0 failed=4 pending=53 cuda1:7713/24564MiB 22% cuda3:6469/24564MiB 98%260812-15:30:24 (UTC+8)
26442026-08-12 15:20:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7369/24564MiB 100%260812-15:20:23 (UTC+8)
26432026-08-12 15:10:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7369/24564MiB 96%260812-15:10:22 (UTC+8)
26422026-08-12 15:00:21 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 93% cuda3:7369/24564MiB 100%260812-15:00:22 (UTC+8)
26412026-08-12 14:50:20 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7369/24564MiB 96%260812-14:50:20 (UTC+8)
26402026-08-12 14:40:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7369/24564MiB 100%260812-14:40:20 (UTC+8)
26392026-08-12 14:30:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7369/24564MiB 100%260812-14:30:19 (UTC+8)
26382026-08-12 14:20:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7369/24564MiB 100%260812-14:20:19 (UTC+8)
26372026-08-12 14:10:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:7349/24564MiB 100%260812-14:10:18 (UTC+8)
26362026-08-12 14:00:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6529/24564MiB 100%260812-14:00:17 (UTC+8)
26352026-08-12 13:50:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6529/24564MiB 99%260812-13:50:16 (UTC+8)
26342026-08-12 13:40:15 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6529/24564MiB 98%260812-13:40:16 (UTC+8)
26332026-08-12 13:30:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6469/24564MiB 100%260812-13:30:15 (UTC+8)
26322026-08-12 13:20:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6469/24564MiB 76%260812-13:20:14 (UTC+8)
26312026-08-12 13:10:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6529/24564MiB 100%260812-13:10:14 (UTC+8)
26302026-08-12 13:00:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6529/24564MiB 98%260812-13:00:13 (UTC+8)
26292026-08-12 12:50:12 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6529/24564MiB 100%260812-12:50:13 (UTC+8)
26282026-08-12 12:40:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6529/24564MiB 100%260812-12:40:12 (UTC+8)
26272026-08-12 12:30:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6789/24564MiB 100%260812-12:30:10 (UTC+8)
26262026-08-12 12:20:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 94% cuda3:6789/24564MiB 97%260812-12:20:10 (UTC+8)
26252026-08-12 12:10:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6789/24564MiB 97%260812-12:10:09 (UTC+8)
26242026-08-12 12:00:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6789/24564MiB 100%260812-12:00:09 (UTC+8)
26232026-08-12 11:50:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 38% cuda3:7349/24564MiB 100%260812-11:50:08 (UTC+8)
26222026-08-12 11:40:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:5445/24564MiB 95%260812-11:40:06 (UTC+8)
26212026-08-12 11:30:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 100%260812-11:30:05 (UTC+8)
26202026-08-12 11:20:03 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 100%260812-11:20:04 (UTC+8)
26192026-08-12 11:10:03 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 95%260812-11:10:03 (UTC+8)
26182026-08-12 11:00:02 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:4905/24564MiB 100%260812-11:00:03 (UTC+8)
26172026-08-12 10:50:01 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:4905/24564MiB 96%260812-10:50:02 (UTC+8)
26162026-08-12 10:40:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4905/24564MiB 100%260812-10:40:01 (UTC+8)
26152026-08-12 10:29:59 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:4605/24564MiB 97%260812-10:30:00 (UTC+8)
26142026-08-12 10:19:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4605/24564MiB 94%260812-10:19:59 (UTC+8)
26132026-08-12 10:09:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:4565/24564MiB 96%260812-10:09:58 (UTC+8)
26122026-08-12 09:59:57 done=216/273 running=0 failed=4 pending=53 cuda1:1637/24564MiB 0% cuda3:4565/24564MiB 95%260812-09:59:58 (UTC+8)
26112026-08-12 09:49:56 done=216/273 running=0 failed=4 pending=53 cuda1:1637/24564MiB 79% cuda3:5445/24564MiB 0%260812-09:49:57 (UTC+8)
26102026-08-12 09:39:56 done=216/273 running=0 failed=4 pending=53 cuda1:1633/24564MiB 77% cuda3:5445/24564MiB 0%260812-09:39:56 (UTC+8)
26092026-08-12 09:29:55 done=216/273 running=0 failed=4 pending=53 cuda1:1435/24564MiB 70% cuda3:5445/24564MiB 0%260812-09:29:56 (UTC+8)
26082026-08-12 09:19:54 done=216/273 running=0 failed=4 pending=53 cuda1:1631/24564MiB 87% cuda3:5445/24564MiB 0%260812-09:19:55 (UTC+8)
26072026-08-12 09:09:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:5445/24564MiB 0%260812-09:09:54 (UTC+8)
26062026-08-12 08:59:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:5445/24564MiB 0%260812-08:59:52 (UTC+8)
26052026-08-12 08:49:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:5445/24564MiB 0%260812-08:49:51 (UTC+8)
26042026-08-12 08:39:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:5445/24564MiB 0%260812-08:39:50 (UTC+8)
26032026-08-12 08:29:49 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:5445/24564MiB 0%260812-08:29:50 (UTC+8)
26022026-08-12 08:19:48 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:5445/24564MiB 100%260812-08:19:49 (UTC+8)
26012026-08-12 08:09:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:5445/24564MiB 100%260812-08:09:47 (UTC+8)
26002026-08-12 07:59:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 0%260812-07:59:47 (UTC+8)
25992026-08-12 07:49:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 100%260812-07:49:46 (UTC+8)
25982026-08-12 07:39:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4885/24564MiB 100%260812-07:39:45 (UTC+8)
25972026-08-12 07:29:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4885/24564MiB 0%260812-07:29:44 (UTC+8)
25962026-08-12 07:19:43 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:4885/24564MiB 0%260812-07:19:44 (UTC+8)
25952026-08-12 07:09:43 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4885/24564MiB 96%260812-07:09:43 (UTC+8)
25942026-08-12 06:59:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4885/24564MiB 0%260812-06:59:43 (UTC+8)
25932026-08-12 06:49:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:4885/24564MiB 96%260812-06:49:42 (UTC+8)
25922026-08-12 06:39:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:4885/24564MiB 0%260812-06:39:42 (UTC+8)
25912026-08-12 06:29:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:4885/24564MiB 0%260812-06:29:41 (UTC+8)
25902026-08-12 06:19:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:4885/24564MiB 0%260812-06:19:40 (UTC+8)
25892026-08-12 06:09:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4885/24564MiB 0%260812-06:09:39 (UTC+8)
25882026-08-12 05:59:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:4885/24564MiB 0%260812-05:59:39 (UTC+8)
25872026-08-12 05:49:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 97%260812-05:49:38 (UTC+8)
25862026-08-12 05:39:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:4625/24564MiB 0%260812-05:39:37 (UTC+8)
25852026-08-12 05:29:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 97%260812-05:29:36 (UTC+8)
25842026-08-12 05:19:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:4625/24564MiB 0%260812-05:19:36 (UTC+8)
25832026-08-12 05:09:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 0%260812-05:09:35 (UTC+8)
25822026-08-12 04:59:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 0%260812-04:59:34 (UTC+8)
25812026-08-12 04:49:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:4625/24564MiB 0%260812-04:49:33 (UTC+8)
25802026-08-12 04:39:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:4625/24564MiB 0%260812-04:39:33 (UTC+8)
25792026-08-12 04:29:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 0%260812-04:29:32 (UTC+8)
25782026-08-12 04:19:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:4625/24564MiB 0%260812-04:19:31 (UTC+8)
25772026-08-12 04:09:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4565/24564MiB 0%260812-04:09:30 (UTC+8)
25762026-08-12 03:59:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:4565/24564MiB 0%260812-03:59:30 (UTC+8)
25752026-08-12 03:49:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4565/24564MiB 0%260812-03:49:29 (UTC+8)
25742026-08-12 03:39:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4565/24564MiB 0%260812-03:39:28 (UTC+8)
25732026-08-12 03:29:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4565/24564MiB 97%260812-03:29:28 (UTC+8)
25722026-08-12 03:19:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4565/24564MiB 0%260812-03:19:27 (UTC+8)
25712026-08-12 03:09:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4565/24564MiB 0%260812-03:09:27 (UTC+8)
25702026-08-12 02:59:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4565/24564MiB 0%260812-02:59:26 (UTC+8)
25692026-08-12 02:49:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:4565/24564MiB 95%260812-02:49:25 (UTC+8)
25682026-08-12 02:39:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:4565/24564MiB 0%260812-02:39:25 (UTC+8)
25672026-08-12 02:29:23 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:5445/24564MiB 95%260812-02:29:24 (UTC+8)
25662026-08-12 02:19:23 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:5445/24564MiB 93%260812-02:19:23 (UTC+8)
25652026-08-12 02:09:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:5445/24564MiB 95%260812-02:09:23 (UTC+8)
25642026-08-12 01:59:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:5445/24564MiB 100%260812-01:59:22 (UTC+8)
25632026-08-12 01:49:21 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 100%260812-01:49:22 (UTC+8)
25622026-08-12 01:39:21 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4885/24564MiB 96%260812-01:39:21 (UTC+8)
25612026-08-12 01:29:20 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4885/24564MiB 96%260812-01:29:20 (UTC+8)
25602026-08-12 01:19:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4885/24564MiB 96%260812-01:19:20 (UTC+8)
25592026-08-12 01:09:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4549/24564MiB 0%260812-01:09:19 (UTC+8)
25582026-08-12 00:59:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:4565/24564MiB 97%260812-00:59:18 (UTC+8)
25572026-08-12 00:49:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:5445/24564MiB 94%260812-00:49:17 (UTC+8)
25562026-08-12 00:39:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 100%260812-00:39:17 (UTC+8)
25552026-08-12 00:29:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:5445/24564MiB 95%260812-00:29:16 (UTC+8)
25542026-08-12 00:19:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 100%260812-00:19:16 (UTC+8)
25532026-08-12 00:09:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:5445/24564MiB 100%260812-00:09:13 (UTC+8)
25522026-08-11 23:59:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4885/24564MiB 96%260811-23:59:11 (UTC+8)
25512026-08-11 23:49:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4885/24564MiB 96%260811-23:49:11 (UTC+8)
25502026-08-11 23:39:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4885/24564MiB 100%260811-23:39:10 (UTC+8)
25492026-08-11 23:29:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:4625/24564MiB 97%260811-23:29:09 (UTC+8)
25482026-08-11 23:19:08 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6529/24564MiB 99%260811-23:19:08 (UTC+8)
25472026-08-11 23:09:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6529/24564MiB 99%260811-23:09:08 (UTC+8)
25462026-08-11 22:59:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6469/24564MiB 100%260811-22:59:06 (UTC+8)
25452026-08-11 22:49:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6469/24564MiB 98%260811-22:49:05 (UTC+8)
25442026-08-11 22:39:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:6469/24564MiB 99%260811-22:39:05 (UTC+8)
25432026-08-11 22:29:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7369/24564MiB 100%260811-22:29:04 (UTC+8)
25422026-08-11 22:19:03 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7369/24564MiB 100%260811-22:19:03 (UTC+8)
25412026-08-11 22:09:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7369/24564MiB 98%260811-22:09:03 (UTC+8)
25402026-08-11 21:59:00 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7349/24564MiB 100%260811-21:59:00 (UTC+8)
25392026-08-11 21:48:59 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7369/24564MiB 99%260811-21:49:00 (UTC+8)
25382026-08-11 21:38:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7369/24564MiB 100%260811-21:38:59 (UTC+8)
25372026-08-11 21:28:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7369/24564MiB 100%260811-21:28:58 (UTC+8)
25362026-08-11 21:18:54 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6469/24564MiB 62%260811-21:18:56 (UTC+8)
25352026-08-11 21:08:53 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6529/24564MiB 98%260811-21:08:54 (UTC+8)
25342026-08-11 20:58:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 99% cuda3:6529/24564MiB 98%260811-20:58:52 (UTC+8)
25332026-08-11 20:48:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:6529/24564MiB 98%260811-20:48:52 (UTC+8)
25322026-08-11 20:38:49 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6469/24564MiB 100%260811-20:38:50 (UTC+8)
25312026-08-11 20:28:48 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6469/24564MiB 98%260811-20:28:49 (UTC+8)
25302026-08-11 20:18:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6529/24564MiB 47%260811-20:18:48 (UTC+8)
25292026-08-11 20:08:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:6529/24564MiB 89%260811-20:08:47 (UTC+8)
25282026-08-11 19:58:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6529/24564MiB 67%260811-19:58:46 (UTC+8)
25272026-08-11 19:48:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6789/24564MiB 100%260811-19:48:46 (UTC+8)
25262026-08-11 19:38:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6789/24564MiB 97%260811-19:38:45 (UTC+8)
25252026-08-11 19:28:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6789/24564MiB 100%260811-19:28:44 (UTC+8)
25242026-08-11 19:18:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6789/24564MiB 100%260811-19:18:42 (UTC+8)
25232026-08-11 19:08:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6789/24564MiB 100%260811-19:08:40 (UTC+8)
25222026-08-11 18:58:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6469/24564MiB 99%260811-18:58:39 (UTC+8)
25212026-08-11 18:48:37 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:3541/24564MiB 85%260811-18:48:37 (UTC+8)
25202026-08-11 18:38:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:2305/24564MiB 49%260811-18:38:37 (UTC+8)
25192026-08-11 18:28:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:3539/24564MiB 99%260811-18:28:36 (UTC+8)
25182026-08-11 18:18:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:2305/24564MiB 45%260811-18:18:35 (UTC+8)
25172026-08-11 18:08:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:3457/24564MiB 98%260811-18:08:34 (UTC+8)
25162026-08-11 17:58:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:3541/24564MiB 99%260811-17:58:33 (UTC+8)
25152026-08-11 17:48:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:3413/24564MiB 99%260811-17:48:32 (UTC+8)
25142026-08-11 17:38:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 96%260811-17:38:32 (UTC+8)
25132026-08-11 17:28:31 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:7349/24564MiB 96%260811-17:28:31 (UTC+8)
25122026-08-11 17:18:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 100%260811-17:18:30 (UTC+8)
25112026-08-11 17:08:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7309/24564MiB 100%260811-17:08:30 (UTC+8)
25102026-08-11 16:58:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 100%260811-16:58:29 (UTC+8)
25092026-08-11 16:48:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 100%260811-16:48:28 (UTC+8)
25082026-08-11 16:38:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7349/24564MiB 100%260811-16:38:27 (UTC+8)
25072026-08-11 16:28:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:6809/24564MiB 98%260811-16:28:27 (UTC+8)
25062026-08-11 16:18:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6789/24564MiB 100%260811-16:18:26 (UTC+8)
25052026-08-11 16:08:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6809/24564MiB 100%260811-16:08:26 (UTC+8)
25042026-08-11 15:58:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6809/24564MiB 99%260811-15:58:25 (UTC+8)
25032026-08-11 15:48:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6509/24564MiB 100%260811-15:48:24 (UTC+8)
25022026-08-11 15:38:21 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6509/24564MiB 98%260811-15:38:23 (UTC+8)
25012026-08-11 15:28:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:6509/24564MiB 100%260811-15:28:21 (UTC+8)
25002026-08-11 15:18:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:6469/24564MiB 100%260811-15:18:19 (UTC+8)
24992026-08-11 15:08:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6469/24564MiB 98%260811-15:08:19 (UTC+8)
24982026-08-11 14:58:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 100%260811-14:58:18 (UTC+8)
24972026-08-11 14:48:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 51%260811-14:48:17 (UTC+8)
24962026-08-11 14:38:15 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7349/24564MiB 100%260811-14:38:16 (UTC+8)
24952026-08-11 14:28:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 6%260811-14:28:15 (UTC+8)
24942026-08-11 14:18:12 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:7349/24564MiB 100%260811-14:18:13 (UTC+8)
24932026-08-11 14:08:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 47%260811-14:08:12 (UTC+8)
24922026-08-11 13:58:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7349/24564MiB 100%260811-13:58:11 (UTC+8)
24912026-08-11 13:48:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:6453/24564MiB 46%260811-13:48:10 (UTC+8)
24902026-08-11 13:38:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7349/24564MiB 100%260811-13:38:10 (UTC+8)
24892026-08-11 13:28:07 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 100%260811-13:28:09 (UTC+8)
24882026-08-11 13:18:07 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 0%260811-13:18:07 (UTC+8)
24872026-08-11 13:08:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 0%260811-13:08:07 (UTC+8)
24862026-08-11 12:58:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:5445/24564MiB 0%260811-12:58:06 (UTC+8)
24852026-08-11 12:48:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:5445/24564MiB 0%260811-12:48:06 (UTC+8)
24842026-08-11 12:38:04 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 100% cuda3:4885/24564MiB 0%260811-12:38:05 (UTC+8)
24832026-08-11 12:28:04 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 96% cuda3:4885/24564MiB 0%260811-12:28:04 (UTC+8)
24822026-08-11 12:18:03 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 99% cuda3:4885/24564MiB 0%260811-12:18:04 (UTC+8)
24812026-08-11 12:08:03 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 100% cuda3:4885/24564MiB 0%260811-12:08:03 (UTC+8)
24802026-08-11 11:58:02 done=216/273 running=0 failed=4 pending=53 cuda1:10479/24564MiB 95% cuda3:6789/24564MiB 97%260811-11:58:03 (UTC+8)
24792026-08-11 11:48:01 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 100% cuda3:6789/24564MiB 100%260811-11:48:02 (UTC+8)
24782026-08-11 11:37:59 done=216/273 running=0 failed=4 pending=53 cuda1:10477/24564MiB 100% cuda3:6789/24564MiB 53%260811-11:38:01 (UTC+8)
24772026-08-11 11:27:58 done=216/273 running=0 failed=4 pending=53 cuda1:5545/24564MiB 100% cuda3:6789/24564MiB 55%260811-11:27:59 (UTC+8)
24762026-08-11 11:17:57 done=216/273 running=0 failed=4 pending=53 cuda1:5545/24564MiB 100% cuda3:6789/24564MiB 100%260811-11:17:58 (UTC+8)
24752026-08-11 11:07:55 done=216/273 running=0 failed=4 pending=53 cuda1:5545/24564MiB 94% cuda3:6789/24564MiB 66%260811-11:07:56 (UTC+8)
24742026-08-11 10:57:55 done=216/273 running=0 failed=4 pending=53 cuda1:9171/24564MiB 100% cuda3:6789/24564MiB 99%260811-10:57:55 (UTC+8)
24732026-08-11 10:47:54 done=216/273 running=0 failed=4 pending=53 cuda1:9171/24564MiB 100% cuda3:6529/24564MiB 100%260811-10:47:55 (UTC+8)
24722026-08-11 10:37:53 done=216/273 running=0 failed=4 pending=53 cuda1:9171/24564MiB 100% cuda3:6529/24564MiB 47%260811-10:37:54 (UTC+8)
24712026-08-11 10:27:51 done=216/273 running=0 failed=4 pending=53 cuda1:9171/24564MiB 100% cuda3:6529/24564MiB 48%260811-10:27:53 (UTC+8)
24702026-08-11 10:17:50 done=216/273 running=0 failed=4 pending=53 cuda1:9171/24564MiB 100% cuda3:6529/24564MiB 47%260811-10:17:51 (UTC+8)
24692026-08-11 10:07:50 done=216/273 running=0 failed=4 pending=53 cuda1:9171/24564MiB 100% cuda3:6529/24564MiB 45%260811-10:07:50 (UTC+8)
24682026-08-11 09:57:49 done=216/273 running=0 failed=4 pending=53 cuda1:9169/24564MiB 100% cuda3:6529/24564MiB 46%260811-09:57:50 (UTC+8)
24672026-08-11 09:47:48 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 100% cuda3:6529/24564MiB 46%260811-09:47:49 (UTC+8)
24662026-08-11 09:37:47 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:6529/24564MiB 43%260811-09:37:48 (UTC+8)
24652026-08-11 09:27:47 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:6529/24564MiB 45%260811-09:27:47 (UTC+8)
24642026-08-11 09:17:46 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:6529/24564MiB 100%260811-09:17:46 (UTC+8)
24632026-08-11 09:07:45 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:6469/24564MiB 46%260811-09:07:46 (UTC+8)
24622026-08-11 08:57:44 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:6469/24564MiB 46%260811-08:57:45 (UTC+8)
24612026-08-11 08:47:43 done=216/273 running=0 failed=4 pending=53 cuda1:4905/24564MiB 99% cuda3:6469/24564MiB 46%260811-08:47:44 (UTC+8)
24602026-08-11 08:37:43 done=216/273 running=0 failed=4 pending=53 cuda1:4905/24564MiB 96% cuda3:6469/24564MiB 45%260811-08:37:43 (UTC+8)
24592026-08-11 08:27:42 done=216/273 running=0 failed=4 pending=53 cuda1:4905/24564MiB 96% cuda3:6469/24564MiB 46%260811-08:27:43 (UTC+8)
24582026-08-11 08:17:41 done=216/273 running=0 failed=4 pending=53 cuda1:4605/24564MiB 97% cuda3:6469/24564MiB 51%260811-08:17:42 (UTC+8)
24572026-08-11 08:07:40 done=216/273 running=0 failed=4 pending=53 cuda1:4605/24564MiB 97% cuda3:6469/24564MiB 44%260811-08:07:41 (UTC+8)
24562026-08-11 07:57:39 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:6469/24564MiB 100%260811-07:57:40 (UTC+8)
24552026-08-11 07:47:39 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 0% cuda3:6469/24564MiB 63%260811-07:47:39 (UTC+8)
24542026-08-11 07:37:38 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 100% cuda3:6469/24564MiB 99%260811-07:37:39 (UTC+8)
24532026-08-11 07:27:38 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 94% cuda3:7349/24564MiB 100%260811-07:27:38 (UTC+8)
24522026-08-11 07:17:37 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 100% cuda3:7349/24564MiB 100%260811-07:17:38 (UTC+8)
24512026-08-11 07:07:36 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 100% cuda3:7349/24564MiB 100%260811-07:07:37 (UTC+8)
24502026-08-11 06:57:36 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7349/24564MiB 100%260811-06:57:36 (UTC+8)
24492026-08-11 06:47:35 done=216/273 running=0 failed=4 pending=53 cuda1:2463/24564MiB 0% cuda3:7349/24564MiB 100%260811-06:47:35 (UTC+8)
24482026-08-11 06:37:34 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 94% cuda3:7349/24564MiB 97%260811-06:37:35 (UTC+8)
24472026-08-11 06:27:33 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 94% cuda3:7349/24564MiB 100%260811-06:27:34 (UTC+8)
24462026-08-11 06:17:32 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 94% cuda3:7349/24564MiB 100%260811-06:17:33 (UTC+8)
24452026-08-11 06:07:31 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 100% cuda3:6789/24564MiB 99%260811-06:07:32 (UTC+8)
24442026-08-11 05:57:31 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:6789/24564MiB 100%260811-05:57:31 (UTC+8)
24432026-08-11 05:47:30 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 96% cuda3:6789/24564MiB 100%260811-05:47:31 (UTC+8)
24422026-08-11 05:37:29 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:6789/24564MiB 100%260811-05:37:30 (UTC+8)
24412026-08-11 05:27:28 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:6469/24564MiB 100%260811-05:27:29 (UTC+8)
24402026-08-11 05:17:28 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:6469/24564MiB 99%260811-05:17:28 (UTC+8)
24392026-08-11 05:07:27 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:6469/24564MiB 99%260811-05:07:28 (UTC+8)
24382026-08-11 04:57:26 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7349/24564MiB 100%260811-04:57:27 (UTC+8)
24372026-08-11 04:47:26 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7349/24564MiB 100%260811-04:47:26 (UTC+8)
24362026-08-11 04:37:25 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7349/24564MiB 100%260811-04:37:25 (UTC+8)
24352026-08-11 04:27:24 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:7349/24564MiB 100%260811-04:27:25 (UTC+8)
24342026-08-11 04:17:24 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7349/24564MiB 100%260811-04:17:24 (UTC+8)
24332026-08-11 04:07:23 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7349/24564MiB 73%260811-04:07:24 (UTC+8)
24322026-08-11 03:57:22 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7349/24564MiB 100%260811-03:57:23 (UTC+8)
24312026-08-11 03:47:22 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:6789/24564MiB 81%260811-03:47:22 (UTC+8)
24302026-08-11 03:37:21 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:6789/24564MiB 97%260811-03:37:22 (UTC+8)
24292026-08-11 03:27:20 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:6789/24564MiB 99%260811-03:27:21 (UTC+8)
24282026-08-11 03:17:20 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:6789/24564MiB 100%260811-03:17:20 (UTC+8)
24272026-08-11 03:07:19 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:6789/24564MiB 100%260811-03:07:19 (UTC+8)
24262026-08-11 02:57:18 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:6529/24564MiB 100%260811-02:57:19 (UTC+8)
24252026-08-11 02:47:18 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 96% cuda3:6529/24564MiB 99%260811-02:47:18 (UTC+8)
24242026-08-11 02:37:17 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:6529/24564MiB 100%260811-02:37:18 (UTC+8)
24232026-08-11 02:27:16 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:6469/24564MiB 100%260811-02:27:17 (UTC+8)
24222026-08-11 02:17:16 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:6469/24564MiB 99%260811-02:17:16 (UTC+8)
24212026-08-11 02:07:15 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7349/24564MiB 100%260811-02:07:16 (UTC+8)
24202026-08-11 01:57:14 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7349/24564MiB 100%260811-01:57:15 (UTC+8)
24192026-08-11 01:47:14 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7349/24564MiB 96%260811-01:47:14 (UTC+8)
24182026-08-11 01:37:13 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7349/24564MiB 98%260811-01:37:13 (UTC+8)
24172026-08-11 01:27:12 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7349/24564MiB 100%260811-01:27:13 (UTC+8)
24162026-08-11 01:17:11 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7349/24564MiB 95%260811-01:17:12 (UTC+8)
24152026-08-11 01:07:11 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7349/24564MiB 100%260811-01:07:11 (UTC+8)
24142026-08-11 00:57:10 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 100% cuda3:7349/24564MiB 96%260811-00:57:11 (UTC+8)
24132026-08-11 00:47:08 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 94% cuda3:6529/24564MiB 98%260811-00:47:10 (UTC+8)
24122026-08-11 00:37:07 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 95% cuda3:6317/24564MiB 42%260811-00:37:08 (UTC+8)
24112026-08-11 00:27:07 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 97% cuda3:6529/24564MiB 100%260811-00:27:07 (UTC+8)
24102026-08-11 00:17:06 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 100% cuda3:6469/24564MiB 100%260811-00:17:06 (UTC+8)
24092026-08-11 00:07:05 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 95% cuda3:6469/24564MiB 98%260811-00:07:06 (UTC+8)
24082026-08-10 23:57:05 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:1917/24564MiB 47%260810-23:57:05 (UTC+8)
24072026-08-10 23:47:04 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:6529/24564MiB 100%260810-23:47:05 (UTC+8)
24062026-08-10 23:37:03 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 94% cuda3:6529/24564MiB 99%260810-23:37:04 (UTC+8)
24052026-08-10 23:27:02 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:6529/24564MiB 100%260810-23:27:03 (UTC+8)
24042026-08-10 23:17:02 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:6789/24564MiB 100%260810-23:17:02 (UTC+8)
24032026-08-10 23:07:01 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:6789/24564MiB 100%260810-23:07:02 (UTC+8)
24022026-08-10 22:57:00 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 100% cuda3:6789/24564MiB 100%260810-22:57:01 (UTC+8)
24012026-08-10 22:47:00 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:6789/24564MiB 97%260810-22:47:00 (UTC+8)
24002026-08-10 22:36:59 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:6789/24564MiB 97%260810-22:36:59 (UTC+8)
23992026-08-10 22:26:57 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:6469/24564MiB 99%260810-22:26:58 (UTC+8)
23982026-08-10 22:16:56 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:7969/24564MiB 100%260810-22:16:57 (UTC+8)
23972026-08-10 22:06:56 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 9% cuda3:7969/24564MiB 100%260810-22:06:56 (UTC+8)
23962026-08-10 21:56:55 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:7969/24564MiB 95%260810-21:56:56 (UTC+8)
23952026-08-10 21:46:55 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:7967/24564MiB 100%260810-21:46:55 (UTC+8)
23942026-08-10 21:36:54 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:401/24564MiB 0%260810-21:36:55 (UTC+8)
23932026-08-10 21:26:53 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 91% cuda3:7851/24564MiB 100%260810-21:26:54 (UTC+8)
23922026-08-10 21:16:52 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 100% cuda3:7851/24564MiB 0%260810-21:16:53 (UTC+8)
23912026-08-10 21:06:50 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:7851/24564MiB 100%260810-21:06:52 (UTC+8)
23902026-08-10 20:56:49 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 94% cuda3:7851/24564MiB 95%260810-20:56:50 (UTC+8)
23892026-08-10 20:46:48 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 94% cuda3:11839/24564MiB 100%260810-20:46:49 (UTC+8)
23882026-08-10 20:26:39 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 19% cuda3:11839/24564MiB 100%260810-20:26:40 (UTC+8)
23872026-08-10 20:16:39 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:11839/24564MiB 100%260810-20:16:39 (UTC+8)
23862026-08-10 19:56:30 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:11839/24564MiB 100%260810-19:56:31 (UTC+8)
23852026-08-10 19:16:05 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:11839/24564MiB 100%260810-19:16:06 (UTC+8)
23842026-08-10 19:06:04 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 95% cuda3:1635/24564MiB 77%260810-19:06:05 (UTC+8)
23832026-08-10 18:56:03 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 98% cuda3:7851/24564MiB 96%260810-18:56:04 (UTC+8)
23822026-08-10 18:46:03 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7851/24564MiB 100%260810-18:46:03 (UTC+8)
23812026-08-10 18:36:02 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7851/24564MiB 100%260810-18:36:03 (UTC+8)
23802026-08-10 18:26:02 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:7851/24564MiB 95%260810-18:26:02 (UTC+8)
23792026-08-10 18:16:00 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:7851/24564MiB 96%260810-18:16:02 (UTC+8)
23782026-08-10 18:05:58 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:7851/24564MiB 95%260810-18:06:00 (UTC+8)
23772026-08-10 17:55:57 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:7851/24564MiB 100%260810-17:55:58 (UTC+8)
23762026-08-10 17:45:57 done=216/273 running=0 failed=4 pending=53 cuda1:4865/24564MiB 100% cuda3:7851/24564MiB 97%260810-17:45:57 (UTC+8)
23752026-08-10 17:35:56 done=216/273 running=0 failed=4 pending=53 cuda1:4605/24564MiB 97% cuda3:7851/24564MiB 100%260810-17:35:57 (UTC+8)
23742026-08-10 17:25:55 done=216/273 running=0 failed=4 pending=53 cuda1:4605/24564MiB 94% cuda3:7851/24564MiB 97%260810-17:25:55 (UTC+8)
23732026-08-10 17:15:53 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:7851/24564MiB 97%260810-17:15:55 (UTC+8)
23722026-08-10 17:05:51 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7689/24564MiB 12%260810-17:05:53 (UTC+8)
23712026-08-10 16:55:51 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7851/24564MiB 97%260810-16:55:52 (UTC+8)
23702026-08-10 16:45:50 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7851/24564MiB 98%260810-16:45:51 (UTC+8)
23692026-08-10 16:25:24 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7851/24564MiB 96%260810-16:25:30 (UTC+8)
23682026-08-10 01:48:06 done=216/273 running=0 failed=4 pending=53 cuda1:4905/24564MiB 97% cuda3:7851/24564MiB 96%260810-01:48:07 (UTC+8)
23672026-08-10 00:37:32 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7851/24564MiB 31%260810-00:37:33 (UTC+8)
23662026-08-10 00:27:31 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:7851/24564MiB 95%260810-00:27:32 (UTC+8)
23652026-08-10 00:07:25 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7851/24564MiB 97%260810-00:07:26 (UTC+8)
23642026-08-09 22:56:46 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:7851/24564MiB 97%260809-22:56:47 (UTC+8)
23632026-08-09 22:46:45 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7851/24564MiB 100%260809-22:46:46 (UTC+8)
23622026-08-09 22:36:38 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7851/24564MiB 96%260809-22:36:45 (UTC+8)
23612026-08-09 22:26:37 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7851/24564MiB 97%260809-22:26:38 (UTC+8)
23602026-08-09 22:16:35 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7851/24564MiB 97%260809-22:16:38 (UTC+8)
23592026-08-09 22:06:34 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7851/24564MiB 96%260809-22:06:35 (UTC+8)
23582026-08-09 21:46:26 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7851/24564MiB 96%260809-21:46:26 (UTC+8)
23572026-08-09 21:36:25 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7851/24564MiB 100%260809-21:36:26 (UTC+8)
23562026-08-09 20:46:01 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7851/24564MiB 0%260809-20:46:02 (UTC+8)
23552026-08-09 20:05:45 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:7851/24564MiB 90%260809-20:05:46 (UTC+8)
23542026-08-09 18:34:59 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 96% cuda3:7851/24564MiB 96%260809-18:35:00 (UTC+8)
23532026-08-09 17:54:40 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:7851/24564MiB 100%260809-17:54:41 (UTC+8)
23522026-08-09 17:04:14 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 95% cuda3:7851/24564MiB 0%260809-17:04:15 (UTC+8)
23512026-08-09 16:03:39 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7851/24564MiB 0%260809-16:03:40 (UTC+8)
23502026-08-09 15:33:28 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:7851/24564MiB 0%260809-15:33:29 (UTC+8)
23492026-08-09 15:13:22 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:7851/24564MiB 97%260809-15:13:23 (UTC+8)
23482026-08-09 15:03:21 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:401/24564MiB 0%260809-15:03:22 (UTC+8)
23472026-08-09 14:33:08 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7851/24564MiB 97%260809-14:33:09 (UTC+8)
23462026-08-09 13:52:52 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7851/24564MiB 0%260809-13:52:53 (UTC+8)
23452026-08-09 13:12:36 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:7851/24564MiB 97%260809-13:12:37 (UTC+8)
23442026-08-09 12:42:25 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:7851/24564MiB 0%260809-12:42:26 (UTC+8)
23432026-08-09 11:41:51 done=216/273 running=0 failed=4 pending=53 cuda1:7967/24564MiB 100% cuda3:7851/24564MiB 0%260809-11:41:52 (UTC+8)
23422026-08-09 11:11:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 100%260809-11:11:41 (UTC+8)
23412026-08-09 10:21:16 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 99% cuda3:7851/24564MiB 96%260809-10:21:17 (UTC+8)
23402026-08-09 10:11:16 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:7851/24564MiB 100%260809-10:11:16 (UTC+8)
23392026-08-09 10:01:15 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:7851/24564MiB 97%260809-10:01:16 (UTC+8)
23382026-08-09 09:20:59 done=216/273 running=0 failed=4 pending=53 cuda1:11839/24564MiB 100% cuda3:7851/24564MiB 97%260809-09:21:00 (UTC+8)
23372026-08-09 08:40:41 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 52% cuda3:7851/24564MiB 100%260809-08:40:42 (UTC+8)
23362026-08-09 08:00:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 95%260809-08:00:26 (UTC+8)
23352026-08-09 07:30:14 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 96%260809-07:30:15 (UTC+8)
23342026-08-09 07:00:03 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 100%260809-07:00:04 (UTC+8)
23332026-08-09 06:50:02 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 97%260809-06:50:03 (UTC+8)
23322026-08-09 06:29:57 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7851/24564MiB 97%260809-06:29:57 (UTC+8)
23312026-08-09 05:59:46 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7851/24564MiB 98%260809-05:59:47 (UTC+8)
23302026-08-09 05:19:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 96%260809-05:19:31 (UTC+8)
23292026-08-09 04:49:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 96%260809-04:49:17 (UTC+8)
23282026-08-09 04:39:15 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 100%260809-04:39:16 (UTC+8)
23272026-08-09 04:09:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 97%260809-04:09:05 (UTC+8)
23262026-08-09 03:59:04 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 97%260809-03:59:04 (UTC+8)
23252026-08-09 03:38:58 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 98%260809-03:38:59 (UTC+8)
23242026-08-09 02:48:32 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 97%260809-02:48:33 (UTC+8)
23232026-08-09 02:38:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 95%260809-02:38:32 (UTC+8)
23222026-08-09 02:28:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 97%260809-02:28:32 (UTC+8)
23212026-08-09 02:08:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 97%260809-02:08:26 (UTC+8)
23202026-08-09 01:48:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 100%260809-01:48:20 (UTC+8)
23192026-08-09 01:28:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:1635/24564MiB 80%260809-01:28:14 (UTC+8)
23182026-08-09 00:37:49 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 96%260809-00:37:50 (UTC+8)
23172026-08-08 23:57:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 96%260808-23:57:33 (UTC+8)
23162026-08-08 23:27:19 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:7851/24564MiB 97%260808-23:27:19 (UTC+8)
23152026-08-08 22:06:31 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 96%260808-22:06:32 (UTC+8)
23142026-08-08 21:46:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 96%260808-21:46:26 (UTC+8)
23132026-08-08 21:26:16 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 100%260808-21:26:21 (UTC+8)
23122026-08-08 21:06:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 96%260808-21:06:11 (UTC+8)
23112026-08-08 20:15:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 0%260808-20:15:43 (UTC+8)
23102026-08-08 18:14:33 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 97%260808-18:14:34 (UTC+8)
23092026-08-08 17:34:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 97%260808-17:34:17 (UTC+8)
23082026-08-08 17:14:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 0%260808-17:14:12 (UTC+8)
23072026-08-08 17:04:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 0%260808-17:04:11 (UTC+8)
23062026-08-08 16:23:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7851/24564MiB 0%260808-16:23:52 (UTC+8)
23052026-08-08 14:42:59 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 71% cuda3:7851/24564MiB 96%260808-14:43:00 (UTC+8)
23042026-08-08 14:22:53 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:7851/24564MiB 0%260808-14:22:53 (UTC+8)
23032026-08-08 13:32:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 95%260808-13:32:29 (UTC+8)
23022026-08-08 12:52:10 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 96%260808-12:52:11 (UTC+8)
23012026-08-08 12:11:54 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 97%260808-12:11:55 (UTC+8)
23002026-08-08 11:11:22 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 95%260808-11:11:23 (UTC+8)
22992026-08-08 10:31:03 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 96%260808-10:31:04 (UTC+8)
22982026-08-08 08:50:05 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7723/24564MiB 13%260808-08:50:06 (UTC+8)
22972026-08-08 07:49:36 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 97%260808-07:49:37 (UTC+8)
22962026-08-08 07:09:18 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7851/24564MiB 100%260808-07:09:21 (UTC+8)
22952026-08-08 06:39:07 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 97%260808-06:39:08 (UTC+8)
22942026-08-08 06:29:02 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:401/24564MiB 0%260808-06:29:07 (UTC+8)
22932026-08-08 05:28:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 95%260808-05:28:36 (UTC+8)
22922026-08-08 05:18:35 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 97%260808-05:18:35 (UTC+8)
22912026-08-08 04:28:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 98%260808-04:28:11 (UTC+8)
22902026-08-08 03:47:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:10479/24564MiB 100%260808-03:47:53 (UTC+8)
22892026-08-08 03:37:52 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:10477/24564MiB 99%260808-03:37:52 (UTC+8)
22882026-08-08 03:27:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:10477/24564MiB 99%260808-03:27:52 (UTC+8)
22872026-08-08 02:57:40 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:5545/24564MiB 95%260808-02:57:41 (UTC+8)
22862026-08-08 02:47:39 done=216/273 running=0 failed=4 pending=53 cuda1:7835/24564MiB 0% cuda3:5545/24564MiB 100%260808-02:47:40 (UTC+8)
22852026-08-08 02:37:39 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:8491/24564MiB 100%260808-02:37:39 (UTC+8)
22842026-08-08 01:47:17 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:8491/24564MiB 99%260808-01:47:18 (UTC+8)
22832026-08-08 01:17:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7851/24564MiB 100%260808-01:17:07 (UTC+8)
22822026-08-08 00:26:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 96%260808-00:26:43 (UTC+8)
22812026-08-07 23:56:30 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 0%260807-23:56:32 (UTC+8)
22802026-08-07 22:55:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 44%260807-22:55:59 (UTC+8)
22792026-08-07 21:55:15 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 100%260807-21:55:20 (UTC+8)
22782026-08-07 21:45:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 0%260807-21:45:15 (UTC+8)
22772026-08-07 19:34:07 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 100%260807-19:34:09 (UTC+8)
22762026-08-07 18:53:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 0%260807-18:53:52 (UTC+8)
22752026-08-07 17:53:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 0%260807-17:53:28 (UTC+8)
22742026-08-07 17:03:01 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7851/24564MiB 0%260807-17:03:02 (UTC+8)
22732026-08-07 16:32:50 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 0%260807-16:32:51 (UTC+8)
22722026-08-07 14:21:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 100%260807-14:21:14 (UTC+8)
22712026-08-07 14:11:09 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 100%260807-14:11:09 (UTC+8)
22702026-08-07 11:59:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 96%260807-11:59:29 (UTC+8)
22692026-08-07 11:49:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 95% cuda3:7851/24564MiB 96%260807-11:49:27 (UTC+8)
22682026-08-07 06:55:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 97%260807-06:55:29 (UTC+8)
22672026-08-07 06:45:26 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 98%260807-06:45:28 (UTC+8)
22662026-08-07 06:15:08 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:7851/24564MiB 97%260807-06:15:09 (UTC+8)
22652026-08-07 05:44:45 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 97%260807-05:44:49 (UTC+8)
22642026-08-07 05:24:29 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 97%260807-05:24:36 (UTC+8)
22632026-08-07 05:14:28 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:7851/24564MiB 97%260807-05:14:29 (UTC+8)
22622026-08-07 05:04:25 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 42%260807-05:04:28 (UTC+8)
22612026-08-07 04:34:06 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:7851/24564MiB 100%260807-04:34:08 (UTC+8)
22602026-08-07 04:13:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 100%260807-04:13:57 (UTC+8)
22592026-08-07 04:03:54 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 96%260807-04:03:56 (UTC+8)
22582026-08-07 03:43:44 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 0% cuda3:7851/24564MiB 97%260807-03:43:46 (UTC+8)
22572026-08-07 03:33:42 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 94% cuda3:7851/24564MiB 45%260807-03:33:44 (UTC+8)
22562026-08-07 01:42:13 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 100%260807-01:42:18 (UTC+8)
22552026-08-07 01:01:47 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 100%260807-01:01:49 (UTC+8)
22542026-08-07 00:41:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 0%260807-00:41:39 (UTC+8)
22532026-08-07 00:01:11 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 0%260807-00:01:13 (UTC+8)
22522026-08-06 23:40:55 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 0%260806-23:41:02 (UTC+8)
22512026-08-06 23:10:38 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 0%260806-23:10:39 (UTC+8)
22502026-08-06 21:49:34 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 98% cuda3:7851/24564MiB 97%260806-21:49:40 (UTC+8)
22492026-08-06 20:58:57 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 100% cuda3:7851/24564MiB 0%260806-20:59:01 (UTC+8)
22482026-08-06 20:48:56 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 96% cuda3:7851/24564MiB 0%260806-20:48:57 (UTC+8)
22472026-08-06 18:57:27 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 0%260806-18:57:32 (UTC+8)
22462026-08-06 18:47:24 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 97% cuda3:7851/24564MiB 0%260806-18:47:27 (UTC+8)
22452026-08-06 16:35:41 done=216/273 running=0 failed=4 pending=53 cuda1:5545/24564MiB 95% cuda3:7851/24564MiB 96%260806-16:35:44 (UTC+8)
22442026-08-06 15:14:40 done=216/273 running=0 failed=4 pending=53 cuda1:8489/24564MiB 100% cuda3:7851/24564MiB 96%260806-15:14:43 (UTC+8)
22432026-08-06 13:53:38 done=216/273 running=0 failed=4 pending=53 cuda1:4865/24564MiB 0% cuda3:7851/24564MiB 96%260806-13:53:41 (UTC+8)
22422026-08-06 13:23:16 done=216/273 running=0 failed=4 pending=53 cuda1:4605/24564MiB 97% cuda3:7851/24564MiB 97%260806-13:23:21 (UTC+8)
22412026-08-06 11:31:52 done=216/273 running=0 failed=4 pending=53 cuda1:5465/24564MiB 96% cuda3:7851/24564MiB 95%260806-11:31:53 (UTC+8)
22402026-08-06 10:41:18 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 72% cuda3:7851/24564MiB 95%260806-10:41:19 (UTC+8)
22392026-08-06 10:00:50 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7851/24564MiB 98%260806-10:00:52 (UTC+8)
22382026-08-06 09:20:23 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7851/24564MiB 97%260806-09:20:25 (UTC+8)
22372026-08-06 08:19:40 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7851/24564MiB 100%260806-08:19:42 (UTC+8)
22362026-08-06 07:59:29 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7851/24564MiB 97%260806-07:59:32 (UTC+8)
22352026-08-06 07:08:55 done=216/273 running=0 failed=4 pending=53 cuda1:6067/24564MiB 0% cuda3:1553/24564MiB 78%260806-07:08:56 (UTC+8)
22342026-08-06 06:38:36 done=216/273 running=0 failed=4 pending=53 cuda1:6067/24564MiB 27% cuda3:7851/24564MiB 96%260806-06:38:38 (UTC+8)
22332026-08-06 06:08:14 done=216/273 running=0 failed=4 pending=53 cuda1:6947/24564MiB 100% cuda3:7851/24564MiB 96%260806-06:08:18 (UTC+8)
22322026-08-06 05:58:10 done=216/273 running=0 failed=4 pending=53 cuda1:6947/24564MiB 100% cuda3:7851/24564MiB 100%260806-05:58:14 (UTC+8)
22312026-08-06 05:48:05 done=216/273 running=0 failed=4 pending=53 cuda1:6947/24564MiB 100% cuda3:7851/24564MiB 100%260806-05:48:10 (UTC+8)
22302026-08-06 05:17:46 done=216/273 running=0 failed=4 pending=53 cuda1:6947/24564MiB 95% cuda3:7851/24564MiB 97%260806-05:17:49 (UTC+8)
22292026-08-06 05:07:40 done=216/273 running=0 failed=4 pending=53 cuda1:6387/24564MiB 100% cuda3:7851/24564MiB 97%260806-05:07:46 (UTC+8)
22282026-08-06 04:27:10 done=216/273 running=0 failed=4 pending=53 cuda1:6067/24564MiB 98% cuda3:7851/24564MiB 2%260806-04:27:15 (UTC+8)
22272026-08-06 04:06:58 done=216/273 running=0 failed=4 pending=53 cuda1:6967/24564MiB 96% cuda3:7851/24564MiB 100%260806-04:07:00 (UTC+8)
22262026-08-06 03:16:20 done=216/273 running=0 failed=4 pending=53 cuda1:6947/24564MiB 100% cuda3:7851/24564MiB 0%260806-03:16:25 (UTC+8)
22252026-08-06 02:56:08 done=216/273 running=0 failed=4 pending=53 cuda1:6387/24564MiB 100% cuda3:7851/24564MiB 0%260806-02:56:12 (UTC+8)
22242026-08-06 02:35:58 done=216/273 running=0 failed=4 pending=53 cuda1:5907/24564MiB 98% cuda3:7851/24564MiB 0%260806-02:36:00 (UTC+8)
22232026-08-06 01:35:15 done=216/273 running=0 failed=4 pending=53 cuda1:6895/24564MiB 95% cuda3:7851/24564MiB 0%260806-01:35:16 (UTC+8)
22222026-08-06 00:44:40 done=216/273 running=0 failed=4 pending=53 cuda1:5855/24564MiB 43% cuda3:7851/24564MiB 97%260806-00:44:41 (UTC+8)
22212026-08-05 23:54:07 done=216/273 running=0 failed=4 pending=53 cuda1:6055/24564MiB 98% cuda3:7851/24564MiB 0%260805-23:54:08 (UTC+8)
22202026-08-05 23:13:37 done=216/273 running=0 failed=4 pending=53 cuda1:6315/24564MiB 99% cuda3:7851/24564MiB 0%260805-23:13:42 (UTC+8)
22192026-08-05 22:33:11 done=216/273 running=0 failed=4 pending=53 cuda1:6875/24564MiB 95% cuda3:7851/24564MiB 97%260805-22:33:11 (UTC+8)
22182026-08-05 21:42:36 done=216/273 running=0 failed=4 pending=53 cuda1:6295/24564MiB 97% cuda3:7851/24564MiB 0%260805-21:42:38 (UTC+8)
22172026-08-05 21:02:07 done=216/273 running=0 failed=4 pending=53 cuda1:5993/24564MiB 98% cuda3:7851/24564MiB 98%260805-21:02:09 (UTC+8)
22162026-08-05 20:11:33 done=216/273 running=0 failed=4 pending=53 cuda1:6961/24564MiB 26% cuda3:7851/24564MiB 96%260805-20:11:34 (UTC+8)
22152026-08-05 19:41:05 done=216/273 running=0 failed=4 pending=53 cuda1:6961/24564MiB 100% cuda3:7851/24564MiB 96%260805-19:41:07 (UTC+8)
22142026-08-05 19:10:48 done=216/273 running=0 failed=4 pending=53 cuda1:6961/24564MiB 25% cuda3:7851/24564MiB 100%260805-19:10:48 (UTC+8)
22132026-08-05 17:59:52 done=216/273 running=0 failed=4 pending=53 cuda1:6401/24564MiB 26% cuda3:7851/24564MiB 90%260805-17:59:57 (UTC+8)
22122026-08-05 16:08:22 done=216/273 running=0 failed=4 pending=53 cuda1:6053/24564MiB 31% cuda3:7851/24564MiB 97%260805-16:08:26 (UTC+8)
22112026-08-05 15:58:17 done=216/273 running=0 failed=4 pending=53 cuda1:6053/24564MiB 27% cuda3:7851/24564MiB 100%260805-15:58:21 (UTC+8)
22102026-08-05 14:37:16 done=216/273 running=0 failed=4 pending=53 cuda1:6105/24564MiB 98% cuda3:7851/24564MiB 97%260805-14:37:18 (UTC+8)
22092026-08-05 13:46:41 done=216/273 running=0 failed=4 pending=53 cuda1:6105/24564MiB 27% cuda3:7851/24564MiB 96%260805-13:46:43 (UTC+8)
22082026-08-05 13:16:20 done=216/273 running=0 failed=4 pending=53 cuda1:6103/24564MiB 29% cuda3:7851/24564MiB 96%260805-13:16:24 (UTC+8)
22072026-08-05 12:45:59 done=216/273 running=0 failed=4 pending=53 cuda1:6781/24564MiB 95% cuda3:7851/24564MiB 94%260805-12:46:03 (UTC+8)
22062026-08-05 11:35:09 done=216/273 running=0 failed=4 pending=53 cuda1:6223/24564MiB 97% cuda3:7851/24564MiB 95%260805-11:35:09 (UTC+8)
22052026-08-05 11:04:51 done=216/273 running=0 failed=4 pending=53 cuda1:6783/24564MiB 100% cuda3:7851/24564MiB 97%260805-11:04:52 (UTC+8)
22042026-08-05 10:44:42 done=216/273 running=0 failed=4 pending=53 cuda1:6761/24564MiB 100% cuda3:7851/24564MiB 97%260805-10:44:43 (UTC+8)
22032026-08-05 10:04:15 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:10479/24564MiB 100%260805-10:04:18 (UTC+8)
22022026-08-05 09:54:15 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:10477/24564MiB 100%260805-09:54:15 (UTC+8)
22012026-08-05 09:44:14 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:10477/24564MiB 100%260805-09:44:15 (UTC+8)
22002026-08-05 09:34:12 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:10477/24564MiB 100%260805-09:34:14 (UTC+8)
21992026-08-05 09:24:08 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 95% cuda3:401/24564MiB 0%260805-09:24:11 (UTC+8)
21982026-08-05 09:14:07 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260805-09:14:08 (UTC+8)
21972026-08-05 09:04:03 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260805-09:04:06 (UTC+8)
21962026-08-05 08:54:01 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260805-08:54:03 (UTC+8)
21952026-08-05 08:33:49 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:4661/24564MiB 26%260805-08:33:52 (UTC+8)
21942026-08-05 08:23:47 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:4661/24564MiB 28%260805-08:23:49 (UTC+8)
21932026-08-05 08:13:46 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:4661/24564MiB 29%260805-08:13:47 (UTC+8)
21922026-08-05 07:53:34 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:4661/24564MiB 30%260805-07:53:38 (UTC+8)
21912026-08-05 07:43:30 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:4661/24564MiB 30%260805-07:43:34 (UTC+8)
21902026-08-05 07:33:29 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:4661/24564MiB 29%260805-07:33:30 (UTC+8)
21892026-08-05 07:23:25 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:4661/24564MiB 29%260805-07:23:29 (UTC+8)
21882026-08-05 07:13:24 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:4661/24564MiB 29%260805-07:13:25 (UTC+8)
21872026-08-05 07:03:23 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:4661/24564MiB 29%260805-07:03:24 (UTC+8)
21862026-08-05 06:53:21 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:4621/24564MiB 29%260805-06:53:23 (UTC+8)
21852026-08-05 06:43:19 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:7817/24564MiB 54%260805-06:43:21 (UTC+8)
21842026-08-05 06:33:18 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:7817/24564MiB 53%260805-06:33:19 (UTC+8)
21832026-08-05 06:23:16 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 15% cuda3:7817/24564MiB 51%260805-06:23:18 (UTC+8)
21822026-08-05 06:13:10 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:7817/24564MiB 54%260805-06:13:15 (UTC+8)
21812026-08-05 06:03:07 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:7817/24564MiB 50%260805-06:03:09 (UTC+8)
21802026-08-05 05:53:04 done=216/273 running=0 failed=4 pending=53 cuda1:1441/24564MiB 74% cuda3:7817/24564MiB 54%260805-05:53:06 (UTC+8)
21792026-08-05 05:42:55 done=216/273 running=0 failed=4 pending=53 cuda1:1439/24564MiB 0% cuda3:7817/24564MiB 54%260805-05:43:04 (UTC+8)
21782026-08-05 05:32:54 done=216/273 running=0 failed=4 pending=53 cuda1:1441/24564MiB 75% cuda3:7817/24564MiB 54%260805-05:32:55 (UTC+8)
21772026-08-05 05:22:51 done=216/273 running=0 failed=4 pending=53 cuda1:1637/24564MiB 85% cuda3:7817/24564MiB 54%260805-05:22:54 (UTC+8)
21762026-08-05 05:02:43 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7817/24564MiB 54%260805-05:02:43 (UTC+8)
21752026-08-05 04:52:39 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:7817/24564MiB 54%260805-04:52:42 (UTC+8)
21742026-08-05 04:42:38 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 93% cuda3:7817/24564MiB 54%260805-04:42:39 (UTC+8)
21732026-08-05 04:32:37 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 93% cuda3:7817/24564MiB 53%260805-04:32:38 (UTC+8)
21722026-08-05 04:22:33 done=216/273 running=0 failed=4 pending=53 cuda1:4905/24564MiB 96% cuda3:7817/24564MiB 54%260805-04:22:37 (UTC+8)
21712026-08-05 04:12:22 done=216/273 running=0 failed=4 pending=53 cuda1:4905/24564MiB 100% cuda3:7817/24564MiB 54%260805-04:12:33 (UTC+8)
21702026-08-05 04:02:19 done=216/273 running=0 failed=4 pending=53 cuda1:401/24564MiB 0% cuda3:7817/24564MiB 53%260805-04:02:22 (UTC+8)
21692026-08-05 03:42:10 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 95% cuda3:7817/24564MiB 54%260805-03:42:10 (UTC+8)
21682026-08-05 03:32:09 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 96% cuda3:7817/24564MiB 53%260805-03:32:10 (UTC+8)
21672026-08-05 03:21:54 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7817/24564MiB 54%260805-03:22:09 (UTC+8)
21662026-08-05 03:11:51 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7817/24564MiB 54%260805-03:11:54 (UTC+8)
21652026-08-05 03:01:49 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7817/24564MiB 54%260805-03:01:51 (UTC+8)
21642026-08-05 02:51:49 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7817/24564MiB 54%260805-02:51:49 (UTC+8)
21632026-08-05 02:31:37 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7817/24564MiB 53%260805-02:31:40 (UTC+8)
21622026-08-05 02:21:35 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7751/24564MiB 54%260805-02:21:37 (UTC+8)
21612026-08-05 02:01:24 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7751/24564MiB 54%260805-02:01:26 (UTC+8)
21602026-08-05 01:51:20 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7751/24564MiB 54%260805-01:51:24 (UTC+8)
21592026-08-05 01:41:18 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7751/24564MiB 54%260805-01:41:20 (UTC+8)
21582026-08-05 01:31:10 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 0% cuda3:7751/24564MiB 82%260805-01:31:18 (UTC+8)
21572026-08-05 01:21:09 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:7751/24564MiB 51%260805-01:21:10 (UTC+8)
21562026-08-05 01:11:08 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:7751/24564MiB 54%260805-01:11:09 (UTC+8)
21552026-08-05 01:01:06 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7751/24564MiB 54%260805-01:01:08 (UTC+8)
21542026-08-05 00:50:58 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:7751/24564MiB 55%260805-00:51:06 (UTC+8)
21532026-08-05 00:40:56 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7751/24564MiB 54%260805-00:40:57 (UTC+8)
21522026-08-05 00:30:55 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7751/24564MiB 54%260805-00:30:56 (UTC+8)
21512026-08-05 00:10:45 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:7751/24564MiB 54%260805-00:10:46 (UTC+8)
21502026-08-05 00:00:44 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7751/24564MiB 54%260805-00:00:45 (UTC+8)
21492026-08-04 23:50:43 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 0% cuda3:7751/24564MiB 86%260804-23:50:44 (UTC+8)
21482026-08-04 23:30:30 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7751/24564MiB 54%260804-23:30:34 (UTC+8)
21472026-08-04 23:20:29 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7751/24564MiB 54%260804-23:20:30 (UTC+8)
21462026-08-04 23:10:21 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7751/24564MiB 51%260804-23:10:29 (UTC+8)
21452026-08-04 23:00:19 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7751/24564MiB 54%260804-23:00:21 (UTC+8)
21442026-08-04 22:50:18 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7751/24564MiB 54%260804-22:50:19 (UTC+8)
21432026-08-04 22:40:17 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7751/24564MiB 53%260804-22:40:18 (UTC+8)
21422026-08-04 22:20:09 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 0% cuda3:7751/24564MiB 54%260804-22:20:09 (UTC+8)
21412026-08-04 22:10:05 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 94% cuda3:7751/24564MiB 54%260804-22:10:09 (UTC+8)
21402026-08-04 21:49:54 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7751/24564MiB 53%260804-21:49:57 (UTC+8)
21392026-08-04 21:39:53 done=216/273 running=0 failed=4 pending=53 cuda1:5903/24564MiB 21% cuda3:7751/24564MiB 54%260804-21:39:54 (UTC+8)
21382026-08-04 21:29:49 done=216/273 running=0 failed=4 pending=53 cuda1:5903/24564MiB 24% cuda3:7751/24564MiB 53%260804-21:29:53 (UTC+8)
21372026-08-04 21:19:46 done=216/273 running=0 failed=4 pending=53 cuda1:5903/24564MiB 21% cuda3:7751/24564MiB 51%260804-21:19:49 (UTC+8)
21362026-08-04 21:09:45 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7751/24564MiB 54%260804-21:09:46 (UTC+8)
21352026-08-04 20:59:43 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7751/24564MiB 53%260804-20:59:45 (UTC+8)
21342026-08-04 20:49:42 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 0% cuda3:7751/24564MiB 55%260804-20:49:43 (UTC+8)
21332026-08-04 20:39:41 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:7751/24564MiB 54%260804-20:39:42 (UTC+8)
21322026-08-04 20:29:40 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7751/24564MiB 54%260804-20:29:41 (UTC+8)
21312026-08-04 20:19:39 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:7751/24564MiB 54%260804-20:19:40 (UTC+8)
21302026-08-04 20:09:39 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:7751/24564MiB 54%260804-20:09:39 (UTC+8)
21292026-08-04 19:59:37 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 94% cuda3:7751/24564MiB 90%260804-19:59:39 (UTC+8)
21282026-08-04 19:39:28 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:7751/24564MiB 54%260804-19:39:29 (UTC+8)
21272026-08-04 19:29:27 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:2419/24564MiB 0%260804-19:29:28 (UTC+8)
21262026-08-04 19:19:26 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:4661/24564MiB 29%260804-19:19:27 (UTC+8)
21252026-08-04 18:59:16 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 93% cuda3:4661/24564MiB 28%260804-18:59:17 (UTC+8)
21242026-08-04 18:39:07 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:4661/24564MiB 27%260804-18:39:07 (UTC+8)
21232026-08-04 18:29:05 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:4661/24564MiB 70%260804-18:29:07 (UTC+8)
21222026-08-04 18:19:02 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:4661/24564MiB 29%260804-18:19:05 (UTC+8)
21212026-08-04 18:09:01 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:4661/24564MiB 28%260804-18:09:02 (UTC+8)
21202026-08-04 17:59:00 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:4661/24564MiB 28%260804-17:59:01 (UTC+8)
21192026-08-04 17:48:59 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 100% cuda3:4661/24564MiB 29%260804-17:49:00 (UTC+8)
21182026-08-04 17:38:57 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 9% cuda3:4661/24564MiB 27%260804-17:38:59 (UTC+8)
21172026-08-04 17:28:57 done=216/273 running=0 failed=4 pending=53 cuda1:4605/24564MiB 97% cuda3:4661/24564MiB 28%260804-17:28:57 (UTC+8)
21162026-08-04 17:18:56 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:4661/24564MiB 26%260804-17:18:57 (UTC+8)
21152026-08-04 17:08:55 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:4661/24564MiB 29%260804-17:08:56 (UTC+8)
21142026-08-04 16:58:54 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:4661/24564MiB 67%260804-16:58:55 (UTC+8)
21132026-08-04 16:48:53 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:4661/24564MiB 27%260804-16:48:54 (UTC+8)
21122026-08-04 16:38:51 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 96% cuda3:4661/24564MiB 29%260804-16:38:53 (UTC+8)
21112026-08-04 16:28:50 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 100% cuda3:4661/24564MiB 27%260804-16:28:51 (UTC+8)
21102026-08-04 16:18:49 done=216/273 running=0 failed=4 pending=53 cuda1:5445/24564MiB 95% cuda3:4661/24564MiB 29%260804-16:18:50 (UTC+8)
21092026-08-04 16:08:48 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:4661/24564MiB 26%260804-16:08:49 (UTC+8)
21082026-08-04 15:48:38 done=216/273 running=0 failed=4 pending=53 cuda1:4565/24564MiB 97% cuda3:4661/24564MiB 27%260804-15:48:39 (UTC+8)
21072026-08-04 15:28:26 done=216/273 running=0 failed=4 pending=53 cuda1:4625/24564MiB 97% cuda3:4661/24564MiB 27%260804-15:28:29 (UTC+8)
21062026-08-04 15:08:16 done=216/273 running=0 failed=4 pending=53 cuda1:1441/24564MiB 74% cuda3:4661/24564MiB 29%260804-15:08:17 (UTC+8)
21052026-08-04 14:58:15 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:4661/24564MiB 29%260804-14:58:16 (UTC+8)
21042026-08-04 14:38:04 done=216/273 running=0 failed=4 pending=53 cuda1:4885/24564MiB 96% cuda3:4661/24564MiB 29%260804-14:38:05 (UTC+8)
21032026-08-04 14:28:03 done=216/273 running=0 failed=4 pending=53 cuda1:7971/24564MiB 93% cuda3:4661/24564MiB 29%260804-14:28:04 (UTC+8)
21022026-08-04 14:07:54 done=216/273 running=0 failed=4 pending=53 cuda1:7967/24564MiB 92% cuda3:4661/24564MiB 29%260804-14:07:55 (UTC+8)
21012026-08-04 13:57:52 done=216/273 running=0 failed=4 pending=53 cuda1:7967/24564MiB 100% cuda3:4661/24564MiB 29%260804-13:57:54 (UTC+8)
21002026-08-04 13:47:51 done=216/273 running=0 failed=4 pending=53 cuda1:7967/24564MiB 100% cuda3:4661/24564MiB 28%260804-13:47:52 (UTC+8)
20992026-08-04 13:27:38 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-13:27:42 (UTC+8)
20982026-08-04 13:17:34 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-13:17:38 (UTC+8)
20972026-08-04 13:07:33 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-13:07:34 (UTC+8)
20962026-08-04 12:57:27 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-12:57:32 (UTC+8)
20952026-08-04 12:27:07 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-12:27:10 (UTC+8)
20942026-08-04 12:17:06 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-12:17:07 (UTC+8)
20932026-08-04 12:07:02 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-12:07:06 (UTC+8)
20922026-08-04 11:46:53 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-11:46:54 (UTC+8)
20912026-08-04 11:36:50 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-11:36:53 (UTC+8)
20902026-08-04 11:26:49 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-11:26:50 (UTC+8)
20892026-08-04 11:16:47 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-11:16:48 (UTC+8)
20882026-08-04 10:56:36 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-10:56:37 (UTC+8)
20872026-08-04 10:46:34 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-10:46:36 (UTC+8)
20862026-08-04 10:36:31 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-10:36:34 (UTC+8)
20852026-08-04 10:26:31 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-10:26:31 (UTC+8)
20842026-08-04 10:06:20 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-10:06:21 (UTC+8)
20832026-08-04 09:56:18 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-09:56:20 (UTC+8)
20822026-08-04 09:46:16 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-09:46:18 (UTC+8)
20812026-08-04 09:36:14 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-09:36:16 (UTC+8)
20802026-08-04 09:26:12 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-09:26:14 (UTC+8)
20792026-08-04 09:16:10 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-09:16:12 (UTC+8)
20782026-08-04 09:06:07 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-09:06:10 (UTC+8)
20772026-08-04 08:56:05 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-08:56:07 (UTC+8)
20762026-08-04 08:46:03 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-08:46:05 (UTC+8)
20752026-08-04 08:36:01 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-08:36:03 (UTC+8)
20742026-08-04 08:25:59 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-08:26:01 (UTC+8)
20732026-08-04 08:05:50 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-08:05:50 (UTC+8)
20722026-08-04 07:55:49 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-07:55:50 (UTC+8)
20712026-08-04 07:45:46 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-07:45:49 (UTC+8)
20702026-08-04 07:35:45 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-07:35:46 (UTC+8)
20692026-08-04 07:15:35 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-07:15:36 (UTC+8)
20682026-08-04 07:05:33 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-07:05:35 (UTC+8)
20672026-08-04 06:45:21 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-06:45:24 (UTC+8)
20662026-08-04 06:35:20 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-06:35:21 (UTC+8)
20652026-08-04 06:25:18 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-06:25:20 (UTC+8)
20642026-08-04 06:15:13 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-06:15:18 (UTC+8)
20632026-08-04 06:05:12 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-06:05:13 (UTC+8)
20622026-08-04 05:55:10 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-05:55:12 (UTC+8)
20612026-08-04 05:45:09 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-05:45:10 (UTC+8)
20602026-08-04 05:24:59 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-05:25:01 (UTC+8)
20592026-08-04 05:14:57 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-05:14:59 (UTC+8)
20582026-08-04 05:04:56 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-05:04:57 (UTC+8)
20572026-08-04 04:54:52 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-04:54:56 (UTC+8)
20562026-08-04 04:44:49 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-04:44:52 (UTC+8)
20552026-08-04 04:34:47 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-04:34:49 (UTC+8)
20542026-08-04 04:14:35 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-04:14:39 (UTC+8)
20532026-08-04 03:54:26 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260804-03:54:27 (UTC+8)
20522026-08-04 03:44:25 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260804-03:44:26 (UTC+8)
20512026-08-04 03:34:24 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 18%260804-03:34:25 (UTC+8)
20502026-08-04 03:24:22 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260804-03:24:24 (UTC+8)
20492026-08-04 03:14:21 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 36%260804-03:14:22 (UTC+8)
20482026-08-04 03:04:18 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 20%260804-03:04:21 (UTC+8)
20472026-08-04 02:54:16 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 20%260804-02:54:18 (UTC+8)
20462026-08-04 02:34:07 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 19%260804-02:34:07 (UTC+8)
20452026-08-04 02:24:05 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 19%260804-02:24:06 (UTC+8)
20442026-08-04 02:14:04 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 72%260804-02:14:05 (UTC+8)
20432026-08-04 02:04:02 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260804-02:04:03 (UTC+8)
20422026-08-04 01:54:01 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260804-01:54:02 (UTC+8)
20412026-08-04 01:43:59 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 20%260804-01:44:01 (UTC+8)
20402026-08-04 01:33:58 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 25%260804-01:33:59 (UTC+8)
20392026-08-04 01:23:56 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 19%260804-01:23:58 (UTC+8)
20382026-08-04 01:13:56 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260804-01:13:56 (UTC+8)
20372026-08-04 01:03:53 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260804-01:03:56 (UTC+8)
20362026-08-04 00:53:50 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260804-00:53:53 (UTC+8)
20352026-08-04 00:33:41 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260804-00:33:42 (UTC+8)
20342026-08-04 00:03:20 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 19%260804-00:03:23 (UTC+8)
20332026-08-03 23:53:18 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 20%260803-23:53:20 (UTC+8)
20322026-08-03 23:22:58 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 20%260803-23:23:00 (UTC+8)
20312026-08-03 23:12:56 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 26%260803-23:12:58 (UTC+8)
20302026-08-03 23:02:55 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 20%260803-23:02:56 (UTC+8)
20292026-08-03 22:52:50 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260803-22:52:55 (UTC+8)
20282026-08-03 22:42:49 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260803-22:42:50 (UTC+8)
20272026-08-03 22:32:47 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 19%260803-22:32:49 (UTC+8)
20262026-08-03 22:12:38 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 20%260803-22:12:39 (UTC+8)
20252026-08-03 22:02:37 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260803-22:02:38 (UTC+8)
20242026-08-03 21:52:36 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 20%260803-21:52:37 (UTC+8)
20232026-08-03 21:42:31 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 19%260803-21:42:36 (UTC+8)
20222026-08-03 21:32:30 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260803-21:32:31 (UTC+8)
20212026-08-03 21:22:27 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 21%260803-21:22:29 (UTC+8)
20202026-08-03 21:12:24 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 19%260803-21:12:27 (UTC+8)
20192026-08-03 20:52:14 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7833/24564MiB 20%260803-20:52:15 (UTC+8)
20182026-08-03 20:32:03 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7827/24564MiB 19%260803-20:32:04 (UTC+8)
20172026-08-03 20:22:02 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:7827/24564MiB 21%260803-20:22:03 (UTC+8)
20162026-08-03 20:01:51 done=216/273 running=0 failed=4 pending=53 cuda1:7851/24564MiB 93% cuda3:7851/24564MiB 95%260803-20:01:53 (UTC+8)
20152026-08-03 19:51:50 done=216/273 running=0 failed=4 pending=53 cuda1:7577/24564MiB 99% cuda3:7851/24564MiB 91%260803-19:51:51 (UTC+8)
20142026-08-03 19:41:49 done=216/273 running=0 failed=4 pending=53 cuda1:1671/24564MiB 85% cuda3:401/24564MiB 0%260803-19:41:50 (UTC+8)
20132026-08-03 19:31:46 done=216/273 running=0 failed=4 pending=53 cuda1:8733/24564MiB 99% cuda3:7719/24564MiB 97%260803-19:31:48 (UTC+8)
20122026-08-03 19:21:45 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-19:21:46 (UTC+8)
20112026-08-03 19:11:44 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-19:11:45 (UTC+8)
20102026-08-03 19:01:37 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-19:01:43 (UTC+8)
20092026-08-03 18:51:35 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-18:51:37 (UTC+8)
20082026-08-03 18:41:33 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-18:41:35 (UTC+8)
20072026-08-03 18:31:30 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-18:31:33 (UTC+8)
20062026-08-03 18:21:27 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-18:21:30 (UTC+8)
20052026-08-03 18:01:17 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-18:01:18 (UTC+8)
20042026-08-03 17:41:06 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-17:41:08 (UTC+8)
20032026-08-03 17:31:03 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-17:31:06 (UTC+8)
20022026-08-03 17:21:01 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-17:21:03 (UTC+8)
20012026-08-03 17:11:00 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-17:11:01 (UTC+8)
20002026-08-03 16:50:51 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-16:50:51 (UTC+8)
19992026-08-03 16:40:46 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-16:40:50 (UTC+8)
19982026-08-03 16:20:36 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-16:20:36 (UTC+8)
19972026-08-03 16:10:33 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-16:10:35 (UTC+8)
19962026-08-03 16:00:27 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-16:00:33 (UTC+8)
19952026-08-03 15:40:17 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-15:40:19 (UTC+8)
19942026-08-03 15:30:09 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-15:30:17 (UTC+8)
19932026-08-03 15:20:03 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-15:20:08 (UTC+8)
19922026-08-03 15:10:02 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-15:10:03 (UTC+8)
19912026-08-03 15:00:00 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-15:00:02 (UTC+8)
19902026-08-03 14:49:59 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-14:50:00 (UTC+8)
19892026-08-03 14:39:57 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-14:39:59 (UTC+8)
19882026-08-03 14:29:56 done=216/273 running=0 failed=4 pending=53 cuda1:14/24564MiB 0% cuda3:14/24564MiB 0%260803-14:29:57 (UTC+8)
#03

Lab-2xA6000

▦ table
last update
260822-00:39:09
records
2000
type
text
latest
CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)
ID数值接收时间
71605CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:39:09 (UTC+8)
71604CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:37:34 (UTC+8)
71603CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:35:59 (UTC+8)
71602CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:34:24 (UTC+8)
71601CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:32:50 (UTC+8)
71600CUDA0: 23GB / 48GB 87% || CUDA1: 23GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:31:15 (UTC+8)
71599CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:29:41 (UTC+8)
71598CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:26:26 (UTC+8)
71597CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:24:52 (UTC+8)
71596CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:23:17 (UTC+8)
71595CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:21:43 (UTC+8)
71594CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:20:08 (UTC+8)
71593CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:18:33 (UTC+8)
71592CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:16:59 (UTC+8)
71591CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:15:23 (UTC+8)
71590CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:13:49 (UTC+8)
71589CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:12:14 (UTC+8)
71588CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:10:39 (UTC+8)
71587CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:09:04 (UTC+8)
71586CUDA0: 23GB / 48GB 95% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:07:29 (UTC+8)
71585CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:05:54 (UTC+8)
71584CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:04:20 (UTC+8)
71583CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:02:45 (UTC+8)
71582CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260822-00:01:10 (UTC+8)
71581CUDA0: 23GB / 48GB 92% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:59:35 (UTC+8)
71580CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:58:00 (UTC+8)
71579CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:56:25 (UTC+8)
71578CUDA0: 23GB / 48GB 92% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:48:14 (UTC+8)
71577CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:46:38 (UTC+8)
71576CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:45:03 (UTC+8)
71575CUDA0: 23GB / 48GB 87% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:43:28 (UTC+8)
71574CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:41:52 (UTC+8)
71573CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:40:17 (UTC+8)
71572CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:38:42 (UTC+8)
71571CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:37:04 (UTC+8)
71570CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 62% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:35:29 (UTC+8)
71569CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:33:53 (UTC+8)
71568CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:32:18 (UTC+8)
71567CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:30:43 (UTC+8)
71566CUDA0: 23GB / 48GB 90% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:29:08 (UTC+8)
71565CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:27:33 (UTC+8)
71564CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:25:58 (UTC+8)
71563CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:24:23 (UTC+8)
71562CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:22:48 (UTC+8)
71561CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:19:34 (UTC+8)
71560CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:17:59 (UTC+8)
71559CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:16:24 (UTC+8)
71558CUDA0: 23GB / 48GB 95% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:14:48 (UTC+8)
71557CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:13:13 (UTC+8)
71556CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:11:38 (UTC+8)
71555CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 82% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:10:03 (UTC+8)
71554CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:08:28 (UTC+8)
71553CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:06:53 (UTC+8)
71552CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:05:18 (UTC+8)
71551CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:03:43 (UTC+8)
71550CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:02:08 (UTC+8)
71549CUDA0: 23GB / 48GB 95% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-23:00:33 (UTC+8)
71548CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:58:58 (UTC+8)
71547CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:57:23 (UTC+8)
71546CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:55:48 (UTC+8)
71545CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:54:13 (UTC+8)
71544CUDA0: 23GB / 48GB 51% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:52:38 (UTC+8)
71543CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:51:03 (UTC+8)
71542CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:49:28 (UTC+8)
71541CUDA0: 23GB / 48GB 90% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:47:53 (UTC+8)
71540CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:46:18 (UTC+8)
71539CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:44:43 (UTC+8)
71538CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:43:08 (UTC+8)
71537CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:41:33 (UTC+8)
71536CUDA0: 23GB / 48GB 95% || CUDA1: 23GB / 48GB 10% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:39:58 (UTC+8)
71535CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:38:23 (UTC+8)
71534CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:36:49 (UTC+8)
71533CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:35:14 (UTC+8)
71532CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:33:39 (UTC+8)
71531CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:32:05 (UTC+8)
71530CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:30:30 (UTC+8)
71529CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:28:55 (UTC+8)
71528CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:27:20 (UTC+8)
71527CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:25:44 (UTC+8)
71526CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:24:09 (UTC+8)
71525CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:22:34 (UTC+8)
71524CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:20:59 (UTC+8)
71523CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:19:24 (UTC+8)
71522CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:17:50 (UTC+8)
71521CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:16:15 (UTC+8)
71520CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:14:40 (UTC+8)
71519CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:13:05 (UTC+8)
71518CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:11:30 (UTC+8)
71517CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:09:55 (UTC+8)
71516CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:08:20 (UTC+8)
71515CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:06:45 (UTC+8)
71514CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:05:10 (UTC+8)
71513CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:03:35 (UTC+8)
71512CUDA0: 23GB / 48GB 93% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:02:00 (UTC+8)
71511CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-22:00:25 (UTC+8)
71510CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:58:50 (UTC+8)
71509CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:57:15 (UTC+8)
71508CUDA0: 23GB / 48GB 93% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:55:40 (UTC+8)
71507CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:54:06 (UTC+8)
71506CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:52:31 (UTC+8)
71505CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:50:56 (UTC+8)
71504CUDA0: 23GB / 48GB 95% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:49:21 (UTC+8)
71503CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:47:46 (UTC+8)
71502CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:46:11 (UTC+8)
71501CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:44:36 (UTC+8)
71500CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:43:01 (UTC+8)
71499CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:41:27 (UTC+8)
71498CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:39:52 (UTC+8)
71497CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:38:17 (UTC+8)
71496CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:36:42 (UTC+8)
71495CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:35:07 (UTC+8)
71494CUDA0: 23GB / 48GB 32% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21346MB | CUDA1 PID:[2633805] 23318MB)260821-21:33:32 (UTC+8)
71493CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:23:43 (UTC+8)
71492CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:22:08 (UTC+8)
71491CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:20:33 (UTC+8)
71490CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:18:58 (UTC+8)
71489CUDA0: 23GB / 48GB 88% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:09:08 (UTC+8)
71488CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:07:33 (UTC+8)
71487CUDA0: 23GB / 48GB 88% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:05:58 (UTC+8)
71486CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-21:04:24 (UTC+8)
71485CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:54:33 (UTC+8)
71484CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:52:58 (UTC+8)
71483CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:51:24 (UTC+8)
71482CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:49:49 (UTC+8)
71481CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:48:14 (UTC+8)
71480CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:46:39 (UTC+8)
71479CUDA0: 23GB / 48GB 93% || CUDA1: 23GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:45:05 (UTC+8)
71478CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:43:30 (UTC+8)
71477CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:41:55 (UTC+8)
71476CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:40:20 (UTC+8)
71475CUDA0: 23GB / 48GB 36% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:37:06 (UTC+8)
71474CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:35:32 (UTC+8)
71473CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:33:57 (UTC+8)
71472CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:32:22 (UTC+8)
71471CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:30:48 (UTC+8)
71470CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:29:13 (UTC+8)
71469CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:27:38 (UTC+8)
71468CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:26:03 (UTC+8)
71467CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:24:28 (UTC+8)
71466CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:22:53 (UTC+8)
71465CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:21:18 (UTC+8)
71464CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:19:43 (UTC+8)
71463CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:18:08 (UTC+8)
71462CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:16:34 (UTC+8)
71461CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:14:59 (UTC+8)
71460CUDA0: 23GB / 48GB 91% || CUDA1: 23GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:13:25 (UTC+8)
71459CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:11:50 (UTC+8)
71458CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:10:15 (UTC+8)
71457CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:08:40 (UTC+8)
71456CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:07:05 (UTC+8)
71455CUDA0: 23GB / 48GB 96% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:05:30 (UTC+8)
71454CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:03:56 (UTC+8)
71453CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:02:21 (UTC+8)
71452CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-20:00:46 (UTC+8)
71451CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:59:11 (UTC+8)
71450CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:57:37 (UTC+8)
71449CUDA0: 23GB / 48GB 87% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:56:02 (UTC+8)
71448CUDA0: 23GB / 48GB 88% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:54:27 (UTC+8)
71447CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:52:52 (UTC+8)
71446CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:51:18 (UTC+8)
71445CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:49:43 (UTC+8)
71444CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:46:29 (UTC+8)
71443CUDA0: 23GB / 48GB 93% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:36:39 (UTC+8)
71442CUDA0: 23GB / 48GB 91% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:35:05 (UTC+8)
71441CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:33:30 (UTC+8)
71440CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:31:55 (UTC+8)
71439CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:30:20 (UTC+8)
71438CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:28:46 (UTC+8)
71437CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:27:11 (UTC+8)
71436CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:25:36 (UTC+8)
71435CUDA0: 23GB / 48GB 98% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:24:01 (UTC+8)
71434CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:22:26 (UTC+8)
71433CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:20:51 (UTC+8)
71432CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:19:17 (UTC+8)
71431CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:17:42 (UTC+8)
71430CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:16:07 (UTC+8)
71429CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:14:32 (UTC+8)
71428CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:12:57 (UTC+8)
71427CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23166MB)260821-19:11:22 (UTC+8)
71426CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:09:48 (UTC+8)
71425CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:08:13 (UTC+8)
71424CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-19:01:41 (UTC+8)
71423CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:58:28 (UTC+8)
71422CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:56:53 (UTC+8)
71421CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:55:18 (UTC+8)
71420CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:53:43 (UTC+8)
71419CUDA0: 23GB / 48GB 0% || CUDA1: 23GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:52:09 (UTC+8)
71418CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:50:34 (UTC+8)
71417CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:48:59 (UTC+8)
71416CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:47:24 (UTC+8)
71415CUDA0: 23GB / 48GB 91% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:44:11 (UTC+8)
71414CUDA0: 23GB / 48GB 38% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:42:36 (UTC+8)
71413CUDA0: 23GB / 48GB 0% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:41:01 (UTC+8)
71412CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:39:27 (UTC+8)
71411CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:37:52 (UTC+8)
71410CUDA0: 23GB / 48GB 20% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:36:18 (UTC+8)
71409CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:34:43 (UTC+8)
71408CUDA0: 23GB / 48GB 99% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:33:08 (UTC+8)
71407CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:31:33 (UTC+8)
71406CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:29:58 (UTC+8)
71405CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:28:24 (UTC+8)
71404CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:26:49 (UTC+8)
71403CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:23:35 (UTC+8)
71402CUDA0: 23GB / 48GB 95% || CUDA1: 23GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:22:00 (UTC+8)
71401CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:20:26 (UTC+8)
71400CUDA0: 23GB / 48GB 86% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:18:51 (UTC+8)
71399CUDA0: 23GB / 48GB 88% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:17:16 (UTC+8)
71398CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:14:02 (UTC+8)
71397CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:12:27 (UTC+8)
71396CUDA0: 23GB / 48GB 81% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:10:52 (UTC+8)
71395CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:09:18 (UTC+8)
71394CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:07:43 (UTC+8)
71393CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:06:08 (UTC+8)
71392CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:04:33 (UTC+8)
71391CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 0% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:02:58 (UTC+8)
71390CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-18:01:24 (UTC+8)
71389CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:59:49 (UTC+8)
71388CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:58:14 (UTC+8)
71387CUDA0: 23GB / 48GB 99% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:56:39 (UTC+8)
71386CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:55:04 (UTC+8)
71385CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:53:29 (UTC+8)
71384CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:43:39 (UTC+8)
71383CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:42:04 (UTC+8)
71382CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:40:29 (UTC+8)
71381CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:38:55 (UTC+8)
71380CUDA0: 23GB / 48GB 88% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:32:23 (UTC+8)
71379CUDA0: 23GB / 48GB 0% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:29:09 (UTC+8)
71378CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:27:34 (UTC+8)
71377CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:26:00 (UTC+8)
71376CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:21:07 (UTC+8)
71375CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:19:32 (UTC+8)
71374CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-17:01:27 (UTC+8)
71373CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:59:52 (UTC+8)
71372CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:58:17 (UTC+8)
71371CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:56:42 (UTC+8)
71370CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:55:07 (UTC+8)
71369CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:53:32 (UTC+8)
71368CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:51:58 (UTC+8)
71367CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:50:23 (UTC+8)
71366CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:48:48 (UTC+8)
71365CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:47:14 (UTC+8)
71364CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:45:39 (UTC+8)
71363CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:44:04 (UTC+8)
71362CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:42:29 (UTC+8)
71361CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:40:55 (UTC+8)
71360CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 20% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23166MB)260821-16:39:20 (UTC+8)
71359CUDA0: 23GB / 48GB 90% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:37:46 (UTC+8)
71358CUDA0: 23GB / 48GB 95% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:36:11 (UTC+8)
71357CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:34:36 (UTC+8)
71356CUDA0: 23GB / 48GB 92% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:33:01 (UTC+8)
71355CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:31:26 (UTC+8)
71354CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:29:51 (UTC+8)
71353CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:28:16 (UTC+8)
71352CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:26:41 (UTC+8)
71351CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:25:07 (UTC+8)
71350CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:23:32 (UTC+8)
71349CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:21:58 (UTC+8)
71348CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:20:23 (UTC+8)
71347CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:18:48 (UTC+8)
71346CUDA0: 23GB / 48GB 88% || CUDA1: 23GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:17:14 (UTC+8)
71345CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:15:39 (UTC+8)
71344CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:14:04 (UTC+8)
71343CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:12:29 (UTC+8)
71342CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:10:54 (UTC+8)
71341CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:09:20 (UTC+8)
71340CUDA0: 23GB / 48GB 88% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:07:45 (UTC+8)
71339CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:06:10 (UTC+8)
71338CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:04:35 (UTC+8)
71337CUDA0: 23GB / 48GB 95% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:03:00 (UTC+8)
71336CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-16:01:26 (UTC+8)
71335CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:59:51 (UTC+8)
71334CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:58:16 (UTC+8)
71333CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:56:42 (UTC+8)
71332CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:55:07 (UTC+8)
71331CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:53:32 (UTC+8)
71330CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:51:57 (UTC+8)
71329CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:50:23 (UTC+8)
71328CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:48:48 (UTC+8)
71327CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:47:13 (UTC+8)
71326CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:45:38 (UTC+8)
71325CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:42:24 (UTC+8)
71324CUDA0: 23GB / 48GB 97% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:40:49 (UTC+8)
71323CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:39:14 (UTC+8)
71322CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:37:40 (UTC+8)
71321CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:36:05 (UTC+8)
71320CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:34:30 (UTC+8)
71319CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:32:55 (UTC+8)
71318CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:31:20 (UTC+8)
71317CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:29:45 (UTC+8)
71316CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:28:10 (UTC+8)
71315CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:26:35 (UTC+8)
71314CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:25:01 (UTC+8)
71313CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23166MB)260821-15:23:26 (UTC+8)
71312CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:21:51 (UTC+8)
71311CUDA0: 23GB / 48GB 95% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:20:16 (UTC+8)
71310CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:18:41 (UTC+8)
71309CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:17:06 (UTC+8)
71308CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:15:32 (UTC+8)
71307CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:13:57 (UTC+8)
71306CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:12:22 (UTC+8)
71305CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-15:10:47 (UTC+8)
71304CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:56:00 (UTC+8)
71303CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:54:25 (UTC+8)
71302CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 6% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:42:56 (UTC+8)
71301CUDA0: 23GB / 48GB 88% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:41:22 (UTC+8)
71300CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:39:47 (UTC+8)
71299CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:36:33 (UTC+8)
71298CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:34:58 (UTC+8)
71297CUDA0: 23GB / 48GB 87% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:25:08 (UTC+8)
71296CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:23:34 (UTC+8)
71295CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:21:59 (UTC+8)
71294CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:20:25 (UTC+8)
71293CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:18:50 (UTC+8)
71292CUDA0: 23GB / 48GB 88% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:17:15 (UTC+8)
71291CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:15:40 (UTC+8)
71290CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:14:05 (UTC+8)
71289CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:12:30 (UTC+8)
71288CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:10:55 (UTC+8)
71287CUDA0: 23GB / 48GB 90% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:09:20 (UTC+8)
71286CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:07:46 (UTC+8)
71285CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:06:11 (UTC+8)
71284CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:04:37 (UTC+8)
71283CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-14:01:22 (UTC+8)
71282CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:59:48 (UTC+8)
71281CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:54:52 (UTC+8)
71280CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 86% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:33:29 (UTC+8)
71279CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:28:37 (UTC+8)
71278CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:27:02 (UTC+8)
71277CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:25:27 (UTC+8)
71276CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:23:52 (UTC+8)
71275CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:20:38 (UTC+8)
71274CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:12:27 (UTC+8)
71273CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-13:10:52 (UTC+8)
71272CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:57:44 (UTC+8)
71271CUDA0: 23GB / 48GB 85% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:54:31 (UTC+8)
71270CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:52:56 (UTC+8)
71269CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:49:42 (UTC+8)
71268CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:48:07 (UTC+8)
71267CUDA0: 23GB / 48GB 90% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:43:14 (UTC+8)
71266CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:41:40 (UTC+8)
71265CUDA0: 23GB / 48GB 91% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:40:05 (UTC+8)
71264CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:38:30 (UTC+8)
71263CUDA0: 23GB / 48GB 89% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:28:40 (UTC+8)
71262CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:25:26 (UTC+8)
71261CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:20:33 (UTC+8)
71260CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-12:12:22 (UTC+8)
71259CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:57:35 (UTC+8)
71258CUDA0: 23GB / 48GB 98% || CUDA1: 23GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:56:00 (UTC+8)
71257CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:54:25 (UTC+8)
71256CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:52:51 (UTC+8)
71255CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:49:37 (UTC+8)
71254CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:48:02 (UTC+8)
71253CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:46:27 (UTC+8)
71252CUDA0: 23GB / 48GB 87% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:43:13 (UTC+8)
71251CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:39:59 (UTC+8)
71250CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:26:51 (UTC+8)
71249CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:18:40 (UTC+8)
71248CUDA0: 23GB / 48GB 93% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:15:26 (UTC+8)
71247CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:13:51 (UTC+8)
71246CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:12:16 (UTC+8)
71245CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:10:41 (UTC+8)
71244CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:09:06 (UTC+8)
71243CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:07:31 (UTC+8)
71242CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:05:56 (UTC+8)
71241CUDA0: 23GB / 48GB 94% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:04:22 (UTC+8)
71240CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:02:47 (UTC+8)
71239CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-11:01:12 (UTC+8)
71238CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:59:37 (UTC+8)
71237CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:58:03 (UTC+8)
71236CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:56:28 (UTC+8)
71235CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:49:56 (UTC+8)
71234CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:45:04 (UTC+8)
71233CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:43:29 (UTC+8)
71232CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:41:54 (UTC+8)
71231CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:30:25 (UTC+8)
71230CUDA0: 23GB / 48GB 100% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:28:50 (UTC+8)
71229CUDA0: 23GB / 48GB 98% || CUDA1: 23GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 23318MB)260821-10:27:15 (UTC+8)
71228CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21496MB | CUDA1 PID:[2633805] 21736MB)260821-10:15:46 (UTC+8)
71227CUDA0: 23GB / 48GB 9% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21348MB | CUDA1 PID:[2633805] 21736MB)260821-10:14:11 (UTC+8)
71226CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-10:12:36 (UTC+8)
71225CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-10:11:01 (UTC+8)
71224CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-10:01:11 (UTC+8)
71223CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:59:36 (UTC+8)
71222CUDA0: 23GB / 48GB 89% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:58:01 (UTC+8)
71221CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:56:26 (UTC+8)
71220CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:54:51 (UTC+8)
71219CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:53:16 (UTC+8)
71218CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:51:41 (UTC+8)
71217CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:50:07 (UTC+8)
71216CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:48:32 (UTC+8)
71215CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:46:57 (UTC+8)
71214CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:45:22 (UTC+8)
71213CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:43:47 (UTC+8)
71212CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:42:12 (UTC+8)
71211CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:40:37 (UTC+8)
71210CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:39:02 (UTC+8)
71209CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 44% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:37:28 (UTC+8)
71208CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:35:53 (UTC+8)
71207CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:34:19 (UTC+8)
71206CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:32:44 (UTC+8)
71205CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:31:09 (UTC+8)
71204CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:29:34 (UTC+8)
71203CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:27:59 (UTC+8)
71202CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:26:25 (UTC+8)
71201CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:24:50 (UTC+8)
71200CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:23:15 (UTC+8)
71199CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:21:40 (UTC+8)
71198CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:20:06 (UTC+8)
71197CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:18:31 (UTC+8)
71196CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:16:56 (UTC+8)
71195CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:15:21 (UTC+8)
71194CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:13:47 (UTC+8)
71193CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:12:12 (UTC+8)
71192CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:10:37 (UTC+8)
71191CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:09:02 (UTC+8)
71190CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:05:48 (UTC+8)
71189CUDA0: 23GB / 48GB 100% || CUDA1: 21GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 21736MB)260821-09:04:14 (UTC+8)
71188CUDA0: 23GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 13266MB)260821-09:02:39 (UTC+8)
71187CUDA0: 23GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 13266MB)260821-09:01:04 (UTC+8)
71186CUDA0: 23GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 21566MB | CUDA1 PID:[2633805] 13266MB)260821-08:59:29 (UTC+8)
71185CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:57:54 (UTC+8)
71184CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:56:19 (UTC+8)
71183CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:54:44 (UTC+8)
71182CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:53:09 (UTC+8)
71181CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:51:34 (UTC+8)
71180CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:50:00 (UTC+8)
71179CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:48:25 (UTC+8)
71178CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:46:50 (UTC+8)
71177CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:45:16 (UTC+8)
71176CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:43:41 (UTC+8)
71175CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:38:48 (UTC+8)
71174CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:33:56 (UTC+8)
71173CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:29:03 (UTC+8)
71172CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12926MB | CUDA1 PID:[2633805] 13266MB)260821-08:27:28 (UTC+8)
71171CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-08:09:23 (UTC+8)
71170CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-08:07:48 (UTC+8)
71169CUDA0: 14GB / 48GB 90% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-08:06:13 (UTC+8)
71168CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:58:02 (UTC+8)
71167CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:53:10 (UTC+8)
71166CUDA0: 14GB / 48GB 89% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:51:35 (UTC+8)
71165CUDA0: 14GB / 48GB 90% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:50:01 (UTC+8)
71164CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:48:26 (UTC+8)
71163CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:46:51 (UTC+8)
71162CUDA0: 14GB / 48GB 90% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:45:16 (UTC+8)
71161CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:43:41 (UTC+8)
71160CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:42:07 (UTC+8)
71159CUDA0: 14GB / 48GB 89% || CUDA1: 13GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:40:32 (UTC+8)
71158CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:38:57 (UTC+8)
71157CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13264MB)260821-07:37:22 (UTC+8)
71156CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13268MB)260821-07:35:48 (UTC+8)
71155CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13268MB)260821-07:34:13 (UTC+8)
71154CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12928MB | CUDA1 PID:[2633805] 13268MB)260821-07:32:39 (UTC+8)
71153CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:31:04 (UTC+8)
71152CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:29:29 (UTC+8)
71151CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:27:54 (UTC+8)
71150CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:26:20 (UTC+8)
71149CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:24:45 (UTC+8)
71148CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:19:52 (UTC+8)
71147CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:15:00 (UTC+8)
71146CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:05:10 (UTC+8)
71145CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:03:35 (UTC+8)
71144CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:02:00 (UTC+8)
71143CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-07:00:26 (UTC+8)
71142CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:58:51 (UTC+8)
71141CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:57:16 (UTC+8)
71140CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:45:47 (UTC+8)
71139CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:44:12 (UTC+8)
71138CUDA0: 14GB / 48GB 90% || CUDA1: 13GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:42:38 (UTC+8)
71137CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:31:09 (UTC+8)
71136CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:29:34 (UTC+8)
71135CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:27:59 (UTC+8)
71134CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:26:25 (UTC+8)
71133CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:24:50 (UTC+8)
71132CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:23:15 (UTC+8)
71131CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 5% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:21:41 (UTC+8)
71130CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:20:06 (UTC+8)
71129CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:18:31 (UTC+8)
71128CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:16:56 (UTC+8)
71127CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:15:21 (UTC+8)
71126CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:13:47 (UTC+8)
71125CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:12:12 (UTC+8)
71124CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13268MB)260821-06:10:37 (UTC+8)
71123CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12930MB | CUDA1 PID:[2633805] 13270MB)260821-06:05:44 (UTC+8)
71122CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-06:00:51 (UTC+8)
71121CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:59:16 (UTC+8)
71120CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:57:41 (UTC+8)
71119CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:44:33 (UTC+8)
71118CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:42:58 (UTC+8)
71117CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:41:23 (UTC+8)
71116CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:39:48 (UTC+8)
71115CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:38:14 (UTC+8)
71114CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:36:39 (UTC+8)
71113CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:35:05 (UTC+8)
71112CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:33:30 (UTC+8)
71111CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:31:55 (UTC+8)
71110CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:30:20 (UTC+8)
71109CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12932MB | CUDA1 PID:[2633805] 13270MB)260821-05:28:45 (UTC+8)
71108CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12886MB | CUDA1 PID:[2633805] 12922MB)260821-05:13:58 (UTC+8)
71107CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12886MB | CUDA1 PID:[2633805] 12922MB)260821-05:12:23 (UTC+8)
71106CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12886MB | CUDA1 PID:[2633805] 12922MB)260821-05:09:09 (UTC+8)
71105CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12886MB | CUDA1 PID:[2633805] 12922MB)260821-05:02:37 (UTC+8)
71104CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12886MB | CUDA1 PID:[2633805] 12922MB)260821-04:52:47 (UTC+8)
71103CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12886MB | CUDA1 PID:[2633805] 12922MB)260821-04:44:36 (UTC+8)
71102CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12886MB | CUDA1 PID:[2633805] 12360MB)260821-04:43:01 (UTC+8)
71101CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12886MB | CUDA1 PID:[2633805] 12360MB)260821-04:41:25 (UTC+8)
71100CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12886MB | CUDA1 PID:[2633805] 12360MB)260821-04:39:51 (UTC+8)
71099CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:38:16 (UTC+8)
71098CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:36:41 (UTC+8)
71097CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:35:06 (UTC+8)
71096CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:33:32 (UTC+8)
71095CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:31:57 (UTC+8)
71094CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:30:22 (UTC+8)
71093CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:28:47 (UTC+8)
71092CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:27:12 (UTC+8)
71091CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:25:38 (UTC+8)
71090CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:24:03 (UTC+8)
71089CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:22:28 (UTC+8)
71088CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:20:54 (UTC+8)
71087CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:16:01 (UTC+8)
71086CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:11:08 (UTC+8)
71085CUDA0: 14GB / 48GB 98% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:09:33 (UTC+8)
71084CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12360MB)260821-04:04:40 (UTC+8)
71083CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12372MB)260821-03:59:48 (UTC+8)
71082CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:58:13 (UTC+8)
71081CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:54:59 (UTC+8)
71080CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:48:28 (UTC+8)
71079CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:46:53 (UTC+8)
71078CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:45:18 (UTC+8)
71077CUDA0: 14GB / 48GB 95% || CUDA1: 12GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:43:43 (UTC+8)
71076CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:42:08 (UTC+8)
71075CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:40:33 (UTC+8)
71074CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:35:40 (UTC+8)
71073CUDA0: 14GB / 48GB 93% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:34:06 (UTC+8)
71072CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:32:31 (UTC+8)
71071CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:30:57 (UTC+8)
71070CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:29:22 (UTC+8)
71069CUDA0: 14GB / 48GB 93% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:27:47 (UTC+8)
71068CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:26:12 (UTC+8)
71067CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12372MB)260821-03:24:37 (UTC+8)
71066CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12370MB)260821-03:23:03 (UTC+8)
71065CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12370MB)260821-03:21:28 (UTC+8)
71064CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12324MB | CUDA1 PID:[2633805] 12370MB)260821-03:19:53 (UTC+8)
71063CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:18:19 (UTC+8)
71062CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:16:44 (UTC+8)
71061CUDA0: 14GB / 48GB 90% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:15:09 (UTC+8)
71060CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:13:34 (UTC+8)
71059CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:11:59 (UTC+8)
71058CUDA0: 14GB / 48GB 89% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:10:25 (UTC+8)
71057CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:08:50 (UTC+8)
71056CUDA0: 14GB / 48GB 99% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:05:36 (UTC+8)
71055CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:04:01 (UTC+8)
71054CUDA0: 14GB / 48GB 93% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:02:27 (UTC+8)
71053CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-03:00:52 (UTC+8)
71052CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:59:17 (UTC+8)
71051CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:57:42 (UTC+8)
71050CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:56:07 (UTC+8)
71049CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:54:32 (UTC+8)
71048CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:52:57 (UTC+8)
71047CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:51:22 (UTC+8)
71046CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:49:48 (UTC+8)
71045CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:48:13 (UTC+8)
71044CUDA0: 14GB / 48GB 93% || CUDA1: 12GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:46:38 (UTC+8)
71043CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12370MB)260821-02:45:03 (UTC+8)
71042CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:33:33 (UTC+8)
71041CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:31:58 (UTC+8)
71040CUDA0: 14GB / 48GB 94% || CUDA1: 12GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:30:23 (UTC+8)
71039CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:28:48 (UTC+8)
71038CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:27:14 (UTC+8)
71037CUDA0: 14GB / 48GB 93% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:25:39 (UTC+8)
71036CUDA0: 14GB / 48GB 95% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:24:04 (UTC+8)
71035CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:22:29 (UTC+8)
71034CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:20:54 (UTC+8)
71033CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:19:20 (UTC+8)
71032CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:17:45 (UTC+8)
71031CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:16:10 (UTC+8)
71030CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:14:35 (UTC+8)
71029CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:13:00 (UTC+8)
71028CUDA0: 14GB / 48GB 94% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:11:25 (UTC+8)
71027CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:09:51 (UTC+8)
71026CUDA0: 14GB / 48GB 94% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:08:16 (UTC+8)
71025CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:06:41 (UTC+8)
71024CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12366MB)260821-02:05:06 (UTC+8)
71023CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12374MB)260821-02:01:53 (UTC+8)
71022CUDA0: 14GB / 48GB 99% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12328MB | CUDA1 PID:[2633805] 12374MB)260821-02:00:18 (UTC+8)
71021CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:57:04 (UTC+8)
71020CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:55:29 (UTC+8)
71019CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:53:55 (UTC+8)
71018CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:49:01 (UTC+8)
71017CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:47:27 (UTC+8)
71016CUDA0: 14GB / 48GB 95% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:45:52 (UTC+8)
71015CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 97% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:44:17 (UTC+8)
71014CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:42:42 (UTC+8)
71013CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:41:07 (UTC+8)
71012CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:39:32 (UTC+8)
71011CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:37:57 (UTC+8)
71010CUDA0: 14GB / 48GB 98% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:36:22 (UTC+8)
71009CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:34:47 (UTC+8)
71008CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:33:12 (UTC+8)
71007CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:31:37 (UTC+8)
71006CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:30:02 (UTC+8)
71005CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:28:28 (UTC+8)
71004CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:26:53 (UTC+8)
71003CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:25:18 (UTC+8)
71002CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 12374MB)260821-01:23:43 (UTC+8)
71001CUDA0: 14GB / 48GB 90% || CUDA1: 11GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 11676MB)260821-01:22:08 (UTC+8)
71000CUDA0: 14GB / 48GB 92% || CUDA1: 11GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12326MB | CUDA1 PID:[2633805] 11676MB)260821-01:20:33 (UTC+8)
70999CUDA0: 13GB / 48GB 100% || CUDA1: 11GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 11676MB)260821-01:18:59 (UTC+8)
70998CUDA0: 13GB / 48GB 100% || CUDA1: 11GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 11676MB)260821-01:17:24 (UTC+8)
70997CUDA0: 13GB / 48GB 100% || CUDA1: 11GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 11676MB)260821-01:15:50 (UTC+8)
70996CUDA0: 13GB / 48GB 100% || CUDA1: 11GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 11676MB)260821-01:14:15 (UTC+8)
70995CUDA0: 13GB / 48GB 92% || CUDA1: 11GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 11676MB)260821-01:12:40 (UTC+8)
70994CUDA0: 13GB / 48GB 93% || CUDA1: 11GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 11676MB)260821-00:47:59 (UTC+8)
70993CUDA0: 13GB / 48GB 100% || CUDA1: 11GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 11676MB)260821-00:46:24 (UTC+8)
70992CUDA0: 13GB / 48GB 100% || CUDA1: 11GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 11676MB)260821-00:44:49 (UTC+8)
70991CUDA0: 13GB / 48GB 100% || CUDA1: 11GB / 48GB 18% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 11676MB)260821-00:43:14 (UTC+8)
70990CUDA0: 13GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 7856MB)260821-00:41:40 (UTC+8)
70989CUDA0: 13GB / 48GB 17% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 11638MB | CUDA1 PID:[2633805] 7856MB)260821-00:40:05 (UTC+8)
70988CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260821-00:38:30 (UTC+8)
70987CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260821-00:36:55 (UTC+8)
70986CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260821-00:35:21 (UTC+8)
70985CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260821-00:33:46 (UTC+8)
70984CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260821-00:32:11 (UTC+8)
70983CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260821-00:30:36 (UTC+8)
70982CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260821-00:29:01 (UTC+8)
70981CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260821-00:27:26 (UTC+8)
70980CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260821-00:17:36 (UTC+8)
70979CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:16:01 (UTC+8)
70978CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:14:26 (UTC+8)
70977CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:12:51 (UTC+8)
70976CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:11:16 (UTC+8)
70975CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:09:41 (UTC+8)
70974CUDA0: 9GB / 48GB 77% || CUDA1: 8GB / 48GB 68% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:08:07 (UTC+8)
70973CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:06:32 (UTC+8)
70972CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:04:57 (UTC+8)
70971CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:03:22 (UTC+8)
70970CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:01:47 (UTC+8)
70969CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260821-00:00:12 (UTC+8)
70968CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-23:55:19 (UTC+8)
70967CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-23:53:44 (UTC+8)
70966CUDA0: 9GB / 48GB 86% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:48:50 (UTC+8)
70965CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:47:15 (UTC+8)
70964CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:45:40 (UTC+8)
70963CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:44:05 (UTC+8)
70962CUDA0: 9GB / 48GB 72% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:42:30 (UTC+8)
70961CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:40:54 (UTC+8)
70960CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:39:19 (UTC+8)
70959CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:37:44 (UTC+8)
70958CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:36:09 (UTC+8)
70957CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:34:34 (UTC+8)
70956CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:32:58 (UTC+8)
70955CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:31:23 (UTC+8)
70954CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:29:48 (UTC+8)
70953CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:28:13 (UTC+8)
70952CUDA0: 9GB / 48GB 69% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:26:37 (UTC+8)
70951CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:25:03 (UTC+8)
70950CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:23:28 (UTC+8)
70949CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:21:52 (UTC+8)
70948CUDA0: 9GB / 48GB 61% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:20:17 (UTC+8)
70947CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:18:42 (UTC+8)
70946CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:17:07 (UTC+8)
70945CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:15:31 (UTC+8)
70944CUDA0: 9GB / 48GB 86% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:13:56 (UTC+8)
70943CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 72% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:02:27 (UTC+8)
70942CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-23:00:51 (UTC+8)
70941CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:59:15 (UTC+8)
70940CUDA0: 9GB / 48GB 78% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:57:40 (UTC+8)
70939CUDA0: 9GB / 48GB 87% || CUDA1: 8GB / 48GB 86% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:56:05 (UTC+8)
70938CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:54:30 (UTC+8)
70937CUDA0: 9GB / 48GB 42% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7610MB | CUDA1 PID:[2633805] 7856MB)260820-22:52:55 (UTC+8)
70936CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:51:19 (UTC+8)
70935CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:49:44 (UTC+8)
70934CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 72% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:48:09 (UTC+8)
70933CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:46:34 (UTC+8)
70932CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:44:59 (UTC+8)
70931CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:43:24 (UTC+8)
70930CUDA0: 9GB / 48GB 57% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:41:49 (UTC+8)
70929CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:40:14 (UTC+8)
70928CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:38:39 (UTC+8)
70927CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:37:04 (UTC+8)
70926CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:35:29 (UTC+8)
70925CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:33:54 (UTC+8)
70924CUDA0: 9GB / 48GB 65% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:32:20 (UTC+8)
70923CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:30:45 (UTC+8)
70922CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7690MB)260820-22:29:10 (UTC+8)
70921CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:27:35 (UTC+8)
70920CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:26:00 (UTC+8)
70919CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:24:25 (UTC+8)
70918CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:22:50 (UTC+8)
70917CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:21:15 (UTC+8)
70916CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:19:40 (UTC+8)
70915CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:18:05 (UTC+8)
70914CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:16:30 (UTC+8)
70913CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:14:55 (UTC+8)
70912CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:13:20 (UTC+8)
70911CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:11:46 (UTC+8)
70910CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:10:10 (UTC+8)
70909CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:08:35 (UTC+8)
70908CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:07:00 (UTC+8)
70907CUDA0: 9GB / 48GB 74% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:05:25 (UTC+8)
70906CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-22:03:50 (UTC+8)
70905CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-22:00:36 (UTC+8)
70904CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:59:01 (UTC+8)
70903CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:57:25 (UTC+8)
70902CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:55:50 (UTC+8)
70901CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:47:39 (UTC+8)
70900CUDA0: 9GB / 48GB 19% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:46:04 (UTC+8)
70899CUDA0: 9GB / 48GB 99% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:44:29 (UTC+8)
70898CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:42:54 (UTC+8)
70897CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:41:20 (UTC+8)
70896CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:39:44 (UTC+8)
70895CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:38:09 (UTC+8)
70894CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:36:34 (UTC+8)
70893CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:34:59 (UTC+8)
70892CUDA0: 9GB / 48GB 84% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:33:24 (UTC+8)
70891CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:31:49 (UTC+8)
70890CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:30:14 (UTC+8)
70889CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:28:39 (UTC+8)
70888CUDA0: 9GB / 48GB 76% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:27:03 (UTC+8)
70887CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:25:28 (UTC+8)
70886CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 97% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:23:53 (UTC+8)
70885CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:22:18 (UTC+8)
70884CUDA0: 9GB / 48GB 99% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:20:43 (UTC+8)
70883CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:19:08 (UTC+8)
70882CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:17:34 (UTC+8)
70881CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:15:59 (UTC+8)
70880CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:14:24 (UTC+8)
70879CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 80% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:12:49 (UTC+8)
70878CUDA0: 9GB / 48GB 84% || CUDA1: 8GB / 48GB 86% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:11:14 (UTC+8)
70877CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-21:09:39 (UTC+8)
70876CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:08:04 (UTC+8)
70875CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-21:06:29 (UTC+8)
70874CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7858MB)260820-21:04:54 (UTC+8)
70873CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7858MB)260820-21:03:19 (UTC+8)
70872CUDA0: 9GB / 48GB 97% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7858MB)260820-21:01:45 (UTC+8)
70871CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7858MB)260820-21:00:10 (UTC+8)
70870CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7858MB)260820-20:56:56 (UTC+8)
70869CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7858MB)260820-20:55:21 (UTC+8)
70868CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 78% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7858MB)260820-20:43:52 (UTC+8)
70867CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7858MB)260820-20:42:17 (UTC+8)
70866CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:30:48 (UTC+8)
70865CUDA0: 9GB / 48GB 11% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:29:13 (UTC+8)
70864CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:27:38 (UTC+8)
70863CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:26:03 (UTC+8)
70862CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:24:28 (UTC+8)
70861CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:22:53 (UTC+8)
70860CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:21:18 (UTC+8)
70859CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:19:43 (UTC+8)
70858CUDA0: 9GB / 48GB 86% || CUDA1: 8GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:18:09 (UTC+8)
70857CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:16:34 (UTC+8)
70856CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:14:59 (UTC+8)
70855CUDA0: 9GB / 48GB 82% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:13:24 (UTC+8)
70854CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:11:49 (UTC+8)
70853CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:10:15 (UTC+8)
70852CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:08:40 (UTC+8)
70851CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:07:06 (UTC+8)
70850CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:05:31 (UTC+8)
70849CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:03:56 (UTC+8)
70848CUDA0: 9GB / 48GB 81% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:02:21 (UTC+8)
70847CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 97% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-20:00:46 (UTC+8)
70846CUDA0: 9GB / 48GB 79% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-19:59:11 (UTC+8)
70845CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-19:57:36 (UTC+8)
70844CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-19:56:02 (UTC+8)
70843CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-19:54:27 (UTC+8)
70842CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-19:52:52 (UTC+8)
70841CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-19:51:18 (UTC+8)
70840CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-19:49:43 (UTC+8)
70839CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:48:08 (UTC+8)
70838CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:46:33 (UTC+8)
70837CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:44:58 (UTC+8)
70836CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:43:23 (UTC+8)
70835CUDA0: 9GB / 48GB 86% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:41:48 (UTC+8)
70834CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:40:14 (UTC+8)
70833CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:38:39 (UTC+8)
70832CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:37:04 (UTC+8)
70831CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:35:29 (UTC+8)
70830CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:33:55 (UTC+8)
70829CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:32:20 (UTC+8)
70828CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:30:45 (UTC+8)
70827CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:29:10 (UTC+8)
70826CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:16:02 (UTC+8)
70825CUDA0: 9GB / 48GB 62% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:14:27 (UTC+8)
70824CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:12:53 (UTC+8)
70823CUDA0: 9GB / 48GB 99% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:11:18 (UTC+8)
70822CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:09:44 (UTC+8)
70821CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:08:09 (UTC+8)
70820CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:06:35 (UTC+8)
70819CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:05:00 (UTC+8)
70818CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:03:25 (UTC+8)
70817CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:01:51 (UTC+8)
70816CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-19:00:16 (UTC+8)
70815CUDA0: 9GB / 48GB 98% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-18:58:41 (UTC+8)
70814CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-18:57:06 (UTC+8)
70813CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-18:55:32 (UTC+8)
70812CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:53:57 (UTC+8)
70811CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:52:22 (UTC+8)
70810CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:50:47 (UTC+8)
70809CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:49:12 (UTC+8)
70808CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 97% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:47:37 (UTC+8)
70807CUDA0: 9GB / 48GB 98% || CUDA1: 8GB / 48GB 86% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:46:03 (UTC+8)
70806CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:44:28 (UTC+8)
70805CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:42:53 (UTC+8)
70804CUDA0: 9GB / 48GB 81% || CUDA1: 8GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:41:19 (UTC+8)
70803CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:39:44 (UTC+8)
70802CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:38:09 (UTC+8)
70801CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:36:34 (UTC+8)
70800CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:34:59 (UTC+8)
70799CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 78% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:33:24 (UTC+8)
70798CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:31:50 (UTC+8)
70797CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:30:15 (UTC+8)
70796CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:28:41 (UTC+8)
70795CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:27:06 (UTC+8)
70794CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7610MB | CUDA1 PID:[2633805] 7856MB)260820-18:25:31 (UTC+8)
70793CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:23:56 (UTC+8)
70792CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:22:21 (UTC+8)
70791CUDA0: 9GB / 48GB 87% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:20:46 (UTC+8)
70790CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:19:12 (UTC+8)
70789CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 82% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:17:37 (UTC+8)
70788CUDA0: 9GB / 48GB 84% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:16:02 (UTC+8)
70787CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:14:27 (UTC+8)
70786CUDA0: 9GB / 48GB 0% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:12:52 (UTC+8)
70785CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:11:17 (UTC+8)
70784CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 74% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:09:42 (UTC+8)
70783CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:08:07 (UTC+8)
70782CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:06:32 (UTC+8)
70781CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:04:57 (UTC+8)
70780CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:03:22 (UTC+8)
70779CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:01:47 (UTC+8)
70778CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-18:00:12 (UTC+8)
70777CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:58:38 (UTC+8)
70776CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:57:03 (UTC+8)
70775CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:55:28 (UTC+8)
70774CUDA0: 9GB / 48GB 0% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:53:53 (UTC+8)
70773CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:52:18 (UTC+8)
70772CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 77% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:50:43 (UTC+8)
70771CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:49:08 (UTC+8)
70770CUDA0: 9GB / 48GB 0% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:47:34 (UTC+8)
70769CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:45:59 (UTC+8)
70768CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:42:45 (UTC+8)
70767CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:41:10 (UTC+8)
70766CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:39:35 (UTC+8)
70765CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:38:00 (UTC+8)
70764CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:36:25 (UTC+8)
70763CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:34:51 (UTC+8)
70762CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:33:16 (UTC+8)
70761CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:31:41 (UTC+8)
70760CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:30:07 (UTC+8)
70759CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:28:32 (UTC+8)
70758CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:26:57 (UTC+8)
70757CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:25:23 (UTC+8)
70756CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:23:48 (UTC+8)
70755CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 38% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:22:13 (UTC+8)
70754CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:20:38 (UTC+8)
70753CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:19:03 (UTC+8)
70752CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:17:28 (UTC+8)
70751CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:15:54 (UTC+8)
70750CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:14:19 (UTC+8)
70749CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:12:45 (UTC+8)
70748CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:11:10 (UTC+8)
70747CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 97% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:06:17 (UTC+8)
70746CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:04:42 (UTC+8)
70745CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-17:01:28 (UTC+8)
70744CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:59:54 (UTC+8)
70743CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:58:19 (UTC+8)
70742CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:56:45 (UTC+8)
70741CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:55:10 (UTC+8)
70740CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:53:35 (UTC+8)
70739CUDA0: 9GB / 48GB 79% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:52:00 (UTC+8)
70738CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:50:25 (UTC+8)
70737CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:48:50 (UTC+8)
70736CUDA0: 9GB / 48GB 98% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:47:15 (UTC+8)
70735CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:45:40 (UTC+8)
70734CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:44:05 (UTC+8)
70733CUDA0: 9GB / 48GB 99% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:42:30 (UTC+8)
70732CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:40:55 (UTC+8)
70731CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:39:21 (UTC+8)
70730CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:37:46 (UTC+8)
70729CUDA0: 9GB / 48GB 87% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:36:11 (UTC+8)
70728CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:34:37 (UTC+8)
70727CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:33:02 (UTC+8)
70726CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:31:27 (UTC+8)
70725CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:29:52 (UTC+8)
70724CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:28:17 (UTC+8)
70723CUDA0: 9GB / 48GB 0% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:25:03 (UTC+8)
70722CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:23:28 (UTC+8)
70721CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:21:54 (UTC+8)
70720CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:18:39 (UTC+8)
70719CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:17:04 (UTC+8)
70718CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:15:29 (UTC+8)
70717CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:13:55 (UTC+8)
70716CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:12:20 (UTC+8)
70715CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:10:46 (UTC+8)
70714CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:09:11 (UTC+8)
70713CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:07:36 (UTC+8)
70712CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:06:01 (UTC+8)
70711CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:04:27 (UTC+8)
70710CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:02:52 (UTC+8)
70709CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-16:01:17 (UTC+8)
70708CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:59:43 (UTC+8)
70707CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:58:08 (UTC+8)
70706CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:56:33 (UTC+8)
70705CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 10% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:54:58 (UTC+8)
70704CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:53:23 (UTC+8)
70703CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:51:48 (UTC+8)
70702CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:48:34 (UTC+8)
70701CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:47:00 (UTC+8)
70700CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:45:25 (UTC+8)
70699CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:43:50 (UTC+8)
70698CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:42:15 (UTC+8)
70697CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:40:41 (UTC+8)
70696CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:39:06 (UTC+8)
70695CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:37:31 (UTC+8)
70694CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:35:57 (UTC+8)
70693CUDA0: 9GB / 48GB 98% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:34:22 (UTC+8)
70692CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:32:47 (UTC+8)
70691CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:31:12 (UTC+8)
70690CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:29:37 (UTC+8)
70689CUDA0: 9GB / 48GB 88% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:28:02 (UTC+8)
70688CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:26:27 (UTC+8)
70687CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:24:52 (UTC+8)
70686CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:23:17 (UTC+8)
70685CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:21:43 (UTC+8)
70684CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:20:08 (UTC+8)
70683CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:18:33 (UTC+8)
70682CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:16:59 (UTC+8)
70681CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:15:24 (UTC+8)
70680CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 83% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:13:49 (UTC+8)
70679CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:12:14 (UTC+8)
70678CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:10:39 (UTC+8)
70677CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:09:04 (UTC+8)
70676CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:07:30 (UTC+8)
70675CUDA0: 9GB / 48GB 80% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:05:55 (UTC+8)
70674CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:04:20 (UTC+8)
70673CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-15:01:06 (UTC+8)
70672CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:59:31 (UTC+8)
70671CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:57:56 (UTC+8)
70670CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:56:22 (UTC+8)
70669CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:53:07 (UTC+8)
70668CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:51:32 (UTC+8)
70667CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 69% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:49:58 (UTC+8)
70666CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 71% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:48:23 (UTC+8)
70665CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:45:09 (UTC+8)
70664CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:43:34 (UTC+8)
70663CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:40:21 (UTC+8)
70662CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:38:46 (UTC+8)
70661CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:37:11 (UTC+8)
70660CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:35:36 (UTC+8)
70659CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:34:01 (UTC+8)
70658CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:32:26 (UTC+8)
70657CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-14:30:52 (UTC+8)
70656CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:29:17 (UTC+8)
70655CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:26:03 (UTC+8)
70654CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:24:28 (UTC+8)
70653CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:22:54 (UTC+8)
70652CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:21:19 (UTC+8)
70651CUDA0: 9GB / 48GB 87% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:19:44 (UTC+8)
70650CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:18:09 (UTC+8)
70649CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:16:34 (UTC+8)
70648CUDA0: 9GB / 48GB 86% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:15:00 (UTC+8)
70647CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:13:25 (UTC+8)
70646CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 80% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:11:50 (UTC+8)
70645CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:10:15 (UTC+8)
70644CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:08:41 (UTC+8)
70643CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:07:05 (UTC+8)
70642CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:05:30 (UTC+8)
70641CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:03:55 (UTC+8)
70640CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:02:21 (UTC+8)
70639CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-14:00:46 (UTC+8)
70638CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:59:11 (UTC+8)
70637CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:57:36 (UTC+8)
70636CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:56:01 (UTC+8)
70635CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:54:26 (UTC+8)
70634CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:52:52 (UTC+8)
70633CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:51:17 (UTC+8)
70632CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:49:42 (UTC+8)
70631CUDA0: 9GB / 48GB 98% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:48:07 (UTC+8)
70630CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:46:32 (UTC+8)
70629CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:44:58 (UTC+8)
70628CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:43:23 (UTC+8)
70627CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:41:48 (UTC+8)
70626CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:40:13 (UTC+8)
70625CUDA0: 9GB / 48GB 83% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:38:38 (UTC+8)
70624CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:37:03 (UTC+8)
70623CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:33:50 (UTC+8)
70622CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:32:15 (UTC+8)
70621CUDA0: 9GB / 48GB 30% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:29:01 (UTC+8)
70620CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:27:26 (UTC+8)
70619CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:25:51 (UTC+8)
70618CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:24:16 (UTC+8)
70617CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:22:41 (UTC+8)
70616CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:21:06 (UTC+8)
70615CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 97% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:19:31 (UTC+8)
70614CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:17:56 (UTC+8)
70613CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:16:21 (UTC+8)
70612CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:14:47 (UTC+8)
70611CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:13:12 (UTC+8)
70610CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:11:37 (UTC+8)
70609CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:10:02 (UTC+8)
70608CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 58% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:08:27 (UTC+8)
70607CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:06:53 (UTC+8)
70606CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:05:18 (UTC+8)
70605CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:03:43 (UTC+8)
70604CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-13:02:08 (UTC+8)
70603CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:58:54 (UTC+8)
70602CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:57:20 (UTC+8)
70601CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:55:45 (UTC+8)
70600CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:54:10 (UTC+8)
70599CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:52:35 (UTC+8)
70598CUDA0: 9GB / 48GB 75% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:51:00 (UTC+8)
70597CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:49:25 (UTC+8)
70596CUDA0: 9GB / 48GB 83% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:47:50 (UTC+8)
70595CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:46:15 (UTC+8)
70594CUDA0: 9GB / 48GB 0% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:44:40 (UTC+8)
70593CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 71% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:43:05 (UTC+8)
70592CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:41:30 (UTC+8)
70591CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:39:56 (UTC+8)
70590CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:38:21 (UTC+8)
70589CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:36:46 (UTC+8)
70588CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:35:11 (UTC+8)
70587CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:33:36 (UTC+8)
70586CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:32:00 (UTC+8)
70585CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-12:30:26 (UTC+8)
70584CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:28:51 (UTC+8)
70583CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:27:16 (UTC+8)
70582CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:25:42 (UTC+8)
70581CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:24:07 (UTC+8)
70580CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:22:33 (UTC+8)
70579CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:19:19 (UTC+8)
70578CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:17:43 (UTC+8)
70577CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:16:09 (UTC+8)
70576CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:14:35 (UTC+8)
70575CUDA0: 9GB / 48GB 99% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:13:00 (UTC+8)
70574CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:11:25 (UTC+8)
70573CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:09:51 (UTC+8)
70572CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:08:16 (UTC+8)
70571CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:06:41 (UTC+8)
70570CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 71% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:05:06 (UTC+8)
70569CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:03:32 (UTC+8)
70568CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:01:57 (UTC+8)
70567CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7858MB)260820-12:00:21 (UTC+8)
70566CUDA0: 9GB / 48GB 85% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 18444MB)260820-11:58:46 (UTC+8)
70565CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:57:12 (UTC+8)
70564CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:55:37 (UTC+8)
70563CUDA0: 20GB / 48GB 94% || CUDA1: 18GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:54:02 (UTC+8)
70562CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:52:27 (UTC+8)
70561CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:50:52 (UTC+8)
70560CUDA0: 20GB / 48GB 94% || CUDA1: 18GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:49:17 (UTC+8)
70559CUDA0: 20GB / 48GB 95% || CUDA1: 18GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:47:42 (UTC+8)
70558CUDA0: 20GB / 48GB 96% || CUDA1: 18GB / 48GB 83% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:46:08 (UTC+8)
70557CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:44:33 (UTC+8)
70556CUDA0: 20GB / 48GB 90% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:42:59 (UTC+8)
70555CUDA0: 20GB / 48GB 95% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:38:06 (UTC+8)
70554CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:34:52 (UTC+8)
70553CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:33:17 (UTC+8)
70552CUDA0: 20GB / 48GB 95% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:31:42 (UTC+8)
70551CUDA0: 20GB / 48GB 89% || CUDA1: 18GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18444MB)260820-11:30:08 (UTC+8)
70550CUDA0: 20GB / 48GB 90% || CUDA1: 18GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18402MB | CUDA1 PID:[2633805] 18446MB)260820-11:28:33 (UTC+8)
70549CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:26:58 (UTC+8)
70548CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:25:23 (UTC+8)
70547CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:23:48 (UTC+8)
70546CUDA0: 20GB / 48GB 94% || CUDA1: 18GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:22:13 (UTC+8)
70545CUDA0: 20GB / 48GB 92% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:20:38 (UTC+8)
70544CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:19:04 (UTC+8)
70543CUDA0: 20GB / 48GB 89% || CUDA1: 18GB / 48GB 61% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:17:29 (UTC+8)
70542CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:15:55 (UTC+8)
70541CUDA0: 20GB / 48GB 90% || CUDA1: 18GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:14:20 (UTC+8)
70540CUDA0: 20GB / 48GB 92% || CUDA1: 18GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:12:45 (UTC+8)
70539CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:11:11 (UTC+8)
70538CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:09:36 (UTC+8)
70537CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:08:01 (UTC+8)
70536CUDA0: 20GB / 48GB 94% || CUDA1: 18GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:06:26 (UTC+8)
70535CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 76% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:04:51 (UTC+8)
70534CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:03:17 (UTC+8)
70533CUDA0: 20GB / 48GB 91% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:01:42 (UTC+8)
70532CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18446MB)260820-11:00:06 (UTC+8)
70531CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18406MB | CUDA1 PID:[2633805] 18426MB)260820-10:58:31 (UTC+8)
70530CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:56:56 (UTC+8)
70529CUDA0: 20GB / 48GB 93% || CUDA1: 18GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:55:21 (UTC+8)
70528CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:53:46 (UTC+8)
70527CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:52:11 (UTC+8)
70526CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:50:36 (UTC+8)
70525CUDA0: 20GB / 48GB 96% || CUDA1: 18GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:49:01 (UTC+8)
70524CUDA0: 20GB / 48GB 94% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:47:27 (UTC+8)
70523CUDA0: 20GB / 48GB 90% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:44:13 (UTC+8)
70522CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:42:38 (UTC+8)
70521CUDA0: 20GB / 48GB 83% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:41:03 (UTC+8)
70520CUDA0: 20GB / 48GB 95% || CUDA1: 18GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:39:28 (UTC+8)
70519CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:37:54 (UTC+8)
70518CUDA0: 20GB / 48GB 95% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:36:19 (UTC+8)
70517CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 16% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:34:44 (UTC+8)
70516CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:33:10 (UTC+8)
70515CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:31:35 (UTC+8)
70514CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18426MB)260820-10:30:00 (UTC+8)
70513CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18386MB | CUDA1 PID:[2633805] 18414MB)260820-10:28:25 (UTC+8)
70512CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:25:12 (UTC+8)
70511CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:23:37 (UTC+8)
70510CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:22:02 (UTC+8)
70509CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:20:27 (UTC+8)
70508CUDA0: 20GB / 48GB 87% || CUDA1: 18GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:18:53 (UTC+8)
70507CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:17:18 (UTC+8)
70506CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:15:43 (UTC+8)
70505CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:14:08 (UTC+8)
70504CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:12:33 (UTC+8)
70503CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:10:59 (UTC+8)
70502CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:09:24 (UTC+8)
70501CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:07:49 (UTC+8)
70500CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:06:14 (UTC+8)
70499CUDA0: 20GB / 48GB 99% || CUDA1: 18GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:04:40 (UTC+8)
70498CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:03:05 (UTC+8)
70497CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-10:01:30 (UTC+8)
70496CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:59:55 (UTC+8)
70495CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:58:21 (UTC+8)
70494CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:56:46 (UTC+8)
70493CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:55:11 (UTC+8)
70492CUDA0: 20GB / 48GB 87% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:53:36 (UTC+8)
70491CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:52:01 (UTC+8)
70490CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:50:26 (UTC+8)
70489CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:48:52 (UTC+8)
70488CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:47:17 (UTC+8)
70487CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:45:42 (UTC+8)
70486CUDA0: 20GB / 48GB 87% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:44:07 (UTC+8)
70485CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:42:33 (UTC+8)
70484CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:40:58 (UTC+8)
70483CUDA0: 20GB / 48GB 95% || CUDA1: 18GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:39:23 (UTC+8)
70482CUDA0: 20GB / 48GB 88% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:37:48 (UTC+8)
70481CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:34:35 (UTC+8)
70480CUDA0: 20GB / 48GB 88% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:33:00 (UTC+8)
70479CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:31:25 (UTC+8)
70478CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:29:50 (UTC+8)
70477CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:28:15 (UTC+8)
70476CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:26:41 (UTC+8)
70475CUDA0: 20GB / 48GB 88% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:25:06 (UTC+8)
70474CUDA0: 20GB / 48GB 88% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:23:31 (UTC+8)
70473CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18376MB | CUDA1 PID:[2633805] 18414MB)260820-09:21:56 (UTC+8)
70472CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18370MB | CUDA1 PID:[2633805] 18408MB)260820-09:20:22 (UTC+8)
70471CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18370MB | CUDA1 PID:[2633805] 18408MB)260820-09:18:47 (UTC+8)
70470CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18370MB | CUDA1 PID:[2633805] 18408MB)260820-09:17:12 (UTC+8)
70469CUDA0: 20GB / 48GB 100% || CUDA1: 18GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18370MB | CUDA1 PID:[2633805] 18400MB)260820-09:13:58 (UTC+8)
70468CUDA0: 20GB / 48GB 16% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 18236MB | CUDA1 PID:[2633805] 7856MB)260820-09:12:23 (UTC+8)
70467CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-09:10:49 (UTC+8)
70466CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-09:09:14 (UTC+8)
70465CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-09:07:39 (UTC+8)
70464CUDA0: 9GB / 48GB 97% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-09:06:04 (UTC+8)
70463CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-09:04:29 (UTC+8)
70462CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-09:02:54 (UTC+8)
70461CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-09:01:19 (UTC+8)
70460CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:59:44 (UTC+8)
70459CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:58:09 (UTC+8)
70458CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:56:35 (UTC+8)
70457CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:55:00 (UTC+8)
70456CUDA0: 9GB / 48GB 0% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:53:25 (UTC+8)
70455CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:51:50 (UTC+8)
70454CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:50:16 (UTC+8)
70453CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:48:41 (UTC+8)
70452CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:47:06 (UTC+8)
70451CUDA0: 9GB / 48GB 99% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:45:31 (UTC+8)
70450CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:43:56 (UTC+8)
70449CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:42:21 (UTC+8)
70448CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:40:47 (UTC+8)
70447CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:37:33 (UTC+8)
70446CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:35:58 (UTC+8)
70445CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:34:23 (UTC+8)
70444CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:32:48 (UTC+8)
70443CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:31:13 (UTC+8)
70442CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:29:39 (UTC+8)
70441CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:28:04 (UTC+8)
70440CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:26:29 (UTC+8)
70439CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:24:55 (UTC+8)
70438CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:23:20 (UTC+8)
70437CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:20:06 (UTC+8)
70436CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:18:31 (UTC+8)
70435CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:16:57 (UTC+8)
70434CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:15:22 (UTC+8)
70433CUDA0: 9GB / 48GB 99% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:13:48 (UTC+8)
70432CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:12:13 (UTC+8)
70431CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:10:38 (UTC+8)
70430CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:09:03 (UTC+8)
70429CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:07:29 (UTC+8)
70428CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:05:54 (UTC+8)
70427CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:04:20 (UTC+8)
70426CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:02:45 (UTC+8)
70425CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-08:01:10 (UTC+8)
70424CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:59:35 (UTC+8)
70423CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:58:00 (UTC+8)
70422CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 65% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:56:25 (UTC+8)
70421CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:54:51 (UTC+8)
70420CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:53:16 (UTC+8)
70419CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:51:41 (UTC+8)
70418CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:50:07 (UTC+8)
70417CUDA0: 9GB / 48GB 80% || CUDA1: 8GB / 48GB 7% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:48:32 (UTC+8)
70416CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:46:57 (UTC+8)
70415CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:45:22 (UTC+8)
70414CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:43:47 (UTC+8)
70413CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:42:12 (UTC+8)
70412CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:40:38 (UTC+8)
70411CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:39:02 (UTC+8)
70410CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:37:27 (UTC+8)
70409CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:35:52 (UTC+8)
70408CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:34:17 (UTC+8)
70407CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:32:43 (UTC+8)
70406CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:31:08 (UTC+8)
70405CUDA0: 9GB / 48GB 87% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:29:33 (UTC+8)
70404CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:27:58 (UTC+8)
70403CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:26:23 (UTC+8)
70402CUDA0: 9GB / 48GB 97% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:24:49 (UTC+8)
70401CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:23:14 (UTC+8)
70400CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:21:39 (UTC+8)
70399CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:20:04 (UTC+8)
70398CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:18:29 (UTC+8)
70397CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:16:55 (UTC+8)
70396CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:15:20 (UTC+8)
70395CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:13:45 (UTC+8)
70394CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:12:10 (UTC+8)
70393CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:10:35 (UTC+8)
70392CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:09:01 (UTC+8)
70391CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:07:26 (UTC+8)
70390CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:05:51 (UTC+8)
70389CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:04:16 (UTC+8)
70388CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:02:42 (UTC+8)
70387CUDA0: 9GB / 48GB 80% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-07:01:07 (UTC+8)
70386CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-06:59:32 (UTC+8)
70385CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-06:57:58 (UTC+8)
70384CUDA0: 9GB / 48GB 0% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7610MB | CUDA1 PID:[2633805] 7856MB)260820-06:56:23 (UTC+8)
70383CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:54:48 (UTC+8)
70382CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:53:14 (UTC+8)
70381CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:51:39 (UTC+8)
70380CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:50:04 (UTC+8)
70379CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:48:29 (UTC+8)
70378CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:46:55 (UTC+8)
70377CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:45:20 (UTC+8)
70376CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:43:45 (UTC+8)
70375CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:42:10 (UTC+8)
70374CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:40:35 (UTC+8)
70373CUDA0: 9GB / 48GB 86% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:39:00 (UTC+8)
70372CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:37:25 (UTC+8)
70371CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:35:50 (UTC+8)
70370CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:34:16 (UTC+8)
70369CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:32:41 (UTC+8)
70368CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:31:06 (UTC+8)
70367CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7610MB | CUDA1 PID:[2633805] 7856MB)260820-06:29:31 (UTC+8)
70366CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:26:18 (UTC+8)
70365CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:24:43 (UTC+8)
70364CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:23:08 (UTC+8)
70363CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:21:33 (UTC+8)
70362CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:19:58 (UTC+8)
70361CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:18:23 (UTC+8)
70360CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:16:48 (UTC+8)
70359CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:15:13 (UTC+8)
70358CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:13:39 (UTC+8)
70357CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:12:04 (UTC+8)
70356CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:10:29 (UTC+8)
70355CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:08:54 (UTC+8)
70354CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:07:19 (UTC+8)
70353CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:05:45 (UTC+8)
70352CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:04:10 (UTC+8)
70351CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:02:35 (UTC+8)
70350CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-06:01:00 (UTC+8)
70349CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-05:59:26 (UTC+8)
70348CUDA0: 9GB / 48GB 88% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-05:57:51 (UTC+8)
70347CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-05:56:16 (UTC+8)
70346CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-05:53:02 (UTC+8)
70345CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260820-05:51:28 (UTC+8)
70344CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:49:53 (UTC+8)
70343CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:48:18 (UTC+8)
70342CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:46:43 (UTC+8)
70341CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:45:08 (UTC+8)
70340CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:43:33 (UTC+8)
70339CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:41:59 (UTC+8)
70338CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:40:24 (UTC+8)
70337CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:38:49 (UTC+8)
70336CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:37:14 (UTC+8)
70335CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:35:40 (UTC+8)
70334CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:30:47 (UTC+8)
70333CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:29:12 (UTC+8)
70332CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:27:37 (UTC+8)
70331CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 0% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:26:02 (UTC+8)
70330CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:24:27 (UTC+8)
70329CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:22:52 (UTC+8)
70328CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:21:17 (UTC+8)
70327CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:19:42 (UTC+8)
70326CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:18:07 (UTC+8)
70325CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:16:32 (UTC+8)
70324CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:14:58 (UTC+8)
70323CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:13:23 (UTC+8)
70322CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:11:48 (UTC+8)
70321CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:10:14 (UTC+8)
70320CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:08:39 (UTC+8)
70319CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:07:05 (UTC+8)
70318CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:05:30 (UTC+8)
70317CUDA0: 9GB / 48GB 97% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:03:55 (UTC+8)
70316CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:02:20 (UTC+8)
70315CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-05:00:45 (UTC+8)
70314CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:59:10 (UTC+8)
70313CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:57:35 (UTC+8)
70312CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:56:01 (UTC+8)
70311CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:54:26 (UTC+8)
70310CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:52:51 (UTC+8)
70309CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:51:16 (UTC+8)
70308CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:49:41 (UTC+8)
70307CUDA0: 9GB / 48GB 97% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:48:06 (UTC+8)
70306CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:46:31 (UTC+8)
70305CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:44:56 (UTC+8)
70304CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:43:21 (UTC+8)
70303CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:41:47 (UTC+8)
70302CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:40:12 (UTC+8)
70301CUDA0: 9GB / 48GB 67% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:38:37 (UTC+8)
70300CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:37:03 (UTC+8)
70299CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:35:28 (UTC+8)
70298CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:33:53 (UTC+8)
70297CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:32:18 (UTC+8)
70296CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:30:44 (UTC+8)
70295CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 84% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:29:09 (UTC+8)
70294CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:27:34 (UTC+8)
70293CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:25:59 (UTC+8)
70292CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:24:24 (UTC+8)
70291CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:22:50 (UTC+8)
70290CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:21:15 (UTC+8)
70289CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:19:40 (UTC+8)
70288CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:18:05 (UTC+8)
70287CUDA0: 9GB / 48GB 76% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:16:30 (UTC+8)
70286CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:14:55 (UTC+8)
70285CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:13:20 (UTC+8)
70284CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:11:46 (UTC+8)
70283CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:10:11 (UTC+8)
70282CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:08:36 (UTC+8)
70281CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:07:01 (UTC+8)
70280CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:05:26 (UTC+8)
70279CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:03:52 (UTC+8)
70278CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-04:00:38 (UTC+8)
70277CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:59:04 (UTC+8)
70276CUDA0: 9GB / 48GB 99% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:57:29 (UTC+8)
70275CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:55:54 (UTC+8)
70274CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:54:19 (UTC+8)
70273CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:52:45 (UTC+8)
70272CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:51:10 (UTC+8)
70271CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:49:35 (UTC+8)
70270CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:48:01 (UTC+8)
70269CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:46:26 (UTC+8)
70268CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:44:51 (UTC+8)
70267CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:43:16 (UTC+8)
70266CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:41:42 (UTC+8)
70265CUDA0: 9GB / 48GB 84% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:40:07 (UTC+8)
70264CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:38:32 (UTC+8)
70263CUDA0: 9GB / 48GB 86% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:36:57 (UTC+8)
70262CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:35:22 (UTC+8)
70261CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:33:47 (UTC+8)
70260CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:32:13 (UTC+8)
70259CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:30:38 (UTC+8)
70258CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:29:03 (UTC+8)
70257CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:27:28 (UTC+8)
70256CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:25:53 (UTC+8)
70255CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 86% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:22:40 (UTC+8)
70254CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:21:05 (UTC+8)
70253CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:19:30 (UTC+8)
70252CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:17:55 (UTC+8)
70251CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:16:21 (UTC+8)
70250CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:14:46 (UTC+8)
70249CUDA0: 9GB / 48GB 88% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:13:11 (UTC+8)
70248CUDA0: 9GB / 48GB 94% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:11:36 (UTC+8)
70247CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:10:01 (UTC+8)
70246CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:08:27 (UTC+8)
70245CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:06:52 (UTC+8)
70244CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:05:17 (UTC+8)
70243CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:03:42 (UTC+8)
70242CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:02:08 (UTC+8)
70241CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-03:00:33 (UTC+8)
70240CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:58:58 (UTC+8)
70239CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:57:23 (UTC+8)
70238CUDA0: 9GB / 48GB 81% || CUDA1: 8GB / 48GB 86% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:55:48 (UTC+8)
70237CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:54:13 (UTC+8)
70236CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:52:38 (UTC+8)
70235CUDA0: 9GB / 48GB 88% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:51:04 (UTC+8)
70234CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:49:29 (UTC+8)
70233CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:47:54 (UTC+8)
70232CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:46:19 (UTC+8)
70231CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:44:44 (UTC+8)
70230CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:43:10 (UTC+8)
70229CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:39:56 (UTC+8)
70228CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:38:21 (UTC+8)
70227CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:36:46 (UTC+8)
70226CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:35:12 (UTC+8)
70225CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:33:37 (UTC+8)
70224CUDA0: 9GB / 48GB 88% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:32:02 (UTC+8)
70223CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:30:27 (UTC+8)
70222CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:28:52 (UTC+8)
70221CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:27:18 (UTC+8)
70220CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:25:43 (UTC+8)
70219CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:24:08 (UTC+8)
70218CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:22:33 (UTC+8)
70217CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 30% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:20:59 (UTC+8)
70216CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:19:24 (UTC+8)
70215CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:17:49 (UTC+8)
70214CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:16:15 (UTC+8)
70213CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:14:40 (UTC+8)
70212CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:13:05 (UTC+8)
70211CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 0% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:11:30 (UTC+8)
70210CUDA0: 9GB / 48GB 35% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:09:56 (UTC+8)
70209CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:08:21 (UTC+8)
70208CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:06:46 (UTC+8)
70207CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:05:11 (UTC+8)
70206CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:03:36 (UTC+8)
70205CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:02:02 (UTC+8)
70204CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-02:00:27 (UTC+8)
70203CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:58:52 (UTC+8)
70202CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:57:17 (UTC+8)
70201CUDA0: 9GB / 48GB 71% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:55:43 (UTC+8)
70200CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:54:08 (UTC+8)
70199CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:52:34 (UTC+8)
70198CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:50:59 (UTC+8)
70197CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:49:25 (UTC+8)
70196CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:47:50 (UTC+8)
70195CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:46:15 (UTC+8)
70194CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 75% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:44:41 (UTC+8)
70193CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:43:06 (UTC+8)
70192CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:41:31 (UTC+8)
70191CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:39:56 (UTC+8)
70190CUDA0: 9GB / 48GB 99% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:38:22 (UTC+8)
70189CUDA0: 9GB / 48GB 86% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:36:47 (UTC+8)
70188CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:35:13 (UTC+8)
70187CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:33:38 (UTC+8)
70186CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:32:03 (UTC+8)
70185CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:30:28 (UTC+8)
70184CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:28:53 (UTC+8)
70183CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:27:18 (UTC+8)
70182CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:25:44 (UTC+8)
70181CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:24:09 (UTC+8)
70180CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:22:34 (UTC+8)
70179CUDA0: 9GB / 48GB 98% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:20:59 (UTC+8)
70178CUDA0: 9GB / 48GB 0% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:19:24 (UTC+8)
70177CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:17:49 (UTC+8)
70176CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:16:14 (UTC+8)
70175CUDA0: 9GB / 48GB 88% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:14:40 (UTC+8)
70174CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 0% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:13:05 (UTC+8)
70173CUDA0: 9GB / 48GB 98% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:11:30 (UTC+8)
70172CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:09:55 (UTC+8)
70171CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 70% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:08:20 (UTC+8)
70170CUDA0: 9GB / 48GB 84% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:06:45 (UTC+8)
70169CUDA0: 9GB / 48GB 87% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:05:11 (UTC+8)
70168CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:03:36 (UTC+8)
70167CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:02:02 (UTC+8)
70166CUDA0: 9GB / 48GB 83% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-01:00:27 (UTC+8)
70165CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:58:52 (UTC+8)
70164CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:57:17 (UTC+8)
70163CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:55:43 (UTC+8)
70162CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:54:08 (UTC+8)
70161CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:52:33 (UTC+8)
70160CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:50:58 (UTC+8)
70159CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:49:24 (UTC+8)
70158CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:47:50 (UTC+8)
70157CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:46:15 (UTC+8)
70156CUDA0: 9GB / 48GB 70% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:44:40 (UTC+8)
70155CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:43:05 (UTC+8)
70154CUDA0: 9GB / 48GB 82% || CUDA1: 8GB / 48GB 82% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:41:30 (UTC+8)
70153CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:39:56 (UTC+8)
70152CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:38:21 (UTC+8)
70151CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:36:46 (UTC+8)
70150CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:35:11 (UTC+8)
70149CUDA0: 9GB / 48GB 80% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:33:36 (UTC+8)
70148CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:32:01 (UTC+8)
70147CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 81% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:30:27 (UTC+8)
70146CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:28:52 (UTC+8)
70145CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:27:18 (UTC+8)
70144CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:25:43 (UTC+8)
70143CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:24:08 (UTC+8)
70142CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:22:33 (UTC+8)
70141CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:20:58 (UTC+8)
70140CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:19:23 (UTC+8)
70139CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:17:48 (UTC+8)
70138CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:16:13 (UTC+8)
70137CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:14:38 (UTC+8)
70136CUDA0: 9GB / 48GB 75% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:13:04 (UTC+8)
70135CUDA0: 9GB / 48GB 75% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:11:29 (UTC+8)
70134CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:09:54 (UTC+8)
70133CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:08:19 (UTC+8)
70132CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:06:44 (UTC+8)
70131CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 85% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:05:10 (UTC+8)
70130CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:01:55 (UTC+8)
70129CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260820-00:00:21 (UTC+8)
70128CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:58:45 (UTC+8)
70127CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:57:10 (UTC+8)
70126CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:55:35 (UTC+8)
70125CUDA0: 9GB / 48GB 0% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:54:00 (UTC+8)
70124CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:52:26 (UTC+8)
70123CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:50:51 (UTC+8)
70122CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:49:17 (UTC+8)
70121CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:47:41 (UTC+8)
70120CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:46:06 (UTC+8)
70119CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:44:31 (UTC+8)
70118CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:42:56 (UTC+8)
70117CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:41:21 (UTC+8)
70116CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:39:46 (UTC+8)
70115CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:38:10 (UTC+8)
70114CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:36:36 (UTC+8)
70113CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:35:01 (UTC+8)
70112CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:33:26 (UTC+8)
70111CUDA0: 9GB / 48GB 92% || CUDA1: 8GB / 48GB 49% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:31:50 (UTC+8)
70110CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:30:15 (UTC+8)
70109CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:28:40 (UTC+8)
70108CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:27:05 (UTC+8)
70107CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 41% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7688MB)260819-23:25:30 (UTC+8)
70106CUDA0: 9GB / 48GB 84% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:23:55 (UTC+8)
70105CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:22:20 (UTC+8)
70104CUDA0: 9GB / 48GB 68% || CUDA1: 8GB / 48GB 83% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:20:45 (UTC+8)
70103CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-23:19:09 (UTC+8)
70102CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:17:34 (UTC+8)
70101CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:16:00 (UTC+8)
70100CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 49% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:14:24 (UTC+8)
70099CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:12:50 (UTC+8)
70098CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:11:15 (UTC+8)
70097CUDA0: 9GB / 48GB 76% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:09:39 (UTC+8)
70096CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:08:04 (UTC+8)
70095CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:06:29 (UTC+8)
70094CUDA0: 9GB / 48GB 88% || CUDA1: 8GB / 48GB 50% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:04:54 (UTC+8)
70093CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:03:19 (UTC+8)
70092CUDA0: 9GB / 48GB 93% || CUDA1: 8GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:01:43 (UTC+8)
70091CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-23:00:08 (UTC+8)
70090CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-22:58:34 (UTC+8)
70089CUDA0: 9GB / 48GB 57% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-22:56:59 (UTC+8)
70088CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-22:55:24 (UTC+8)
70087CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-22:53:50 (UTC+8)
70086CUDA0: 9GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7778MB | CUDA1 PID:[2633805] 7856MB)260819-22:52:15 (UTC+8)
70085CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:50:41 (UTC+8)
70084CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:49:06 (UTC+8)
70083CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:47:31 (UTC+8)
70082CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:45:56 (UTC+8)
70081CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:44:21 (UTC+8)
70080CUDA0: 9GB / 48GB 77% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:42:46 (UTC+8)
70079CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 72% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:41:11 (UTC+8)
70078CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:39:37 (UTC+8)
70077CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 85% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:38:02 (UTC+8)
70076CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:36:27 (UTC+8)
70075CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:34:52 (UTC+8)
70074CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 86% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:33:17 (UTC+8)
70073CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:31:42 (UTC+8)
70072CUDA0: 9GB / 48GB 96% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:30:07 (UTC+8)
70071CUDA0: 9GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:28:33 (UTC+8)
70070CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:26:58 (UTC+8)
70069CUDA0: 9GB / 48GB 100% || CUDA1: 8GB / 48GB 74% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:25:23 (UTC+8)
70068CUDA0: 9GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:23:48 (UTC+8)
70067CUDA0: 9GB / 48GB 90% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 7776MB | CUDA1 PID:[2633805] 7856MB)260819-22:22:14 (UTC+8)
70066CUDA0: 14GB / 48GB 92% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:20:39 (UTC+8)
70065CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:19:04 (UTC+8)
70064CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:17:30 (UTC+8)
70063CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:15:55 (UTC+8)
70062CUDA0: 14GB / 48GB 98% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:14:20 (UTC+8)
70061CUDA0: 14GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:12:45 (UTC+8)
70060CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:11:10 (UTC+8)
70059CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:09:35 (UTC+8)
70058CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:08:00 (UTC+8)
70057CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:04:46 (UTC+8)
70056CUDA0: 14GB / 48GB 90% || CUDA1: 8GB / 48GB 41% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:03:12 (UTC+8)
70055CUDA0: 14GB / 48GB 73% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:01:37 (UTC+8)
70054CUDA0: 14GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-22:00:02 (UTC+8)
70053CUDA0: 14GB / 48GB 51% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:58:27 (UTC+8)
70052CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:56:52 (UTC+8)
70051CUDA0: 14GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:55:17 (UTC+8)
70050CUDA0: 14GB / 48GB 97% || CUDA1: 8GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:53:42 (UTC+8)
70049CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:52:07 (UTC+8)
70048CUDA0: 14GB / 48GB 93% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:50:32 (UTC+8)
70047CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 83% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:48:58 (UTC+8)
70046CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:47:23 (UTC+8)
70045CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:45:48 (UTC+8)
70044CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:44:13 (UTC+8)
70043CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:42:38 (UTC+8)
70042CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:41:04 (UTC+8)
70041CUDA0: 14GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:39:29 (UTC+8)
70040CUDA0: 14GB / 48GB 38% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:37:54 (UTC+8)
70039CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 79% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:36:19 (UTC+8)
70038CUDA0: 14GB / 48GB 92% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:34:44 (UTC+8)
70037CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:33:09 (UTC+8)
70036CUDA0: 14GB / 48GB 84% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:31:34 (UTC+8)
70035CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:30:00 (UTC+8)
70034CUDA0: 14GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:28:25 (UTC+8)
70033CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:26:51 (UTC+8)
70032CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:25:16 (UTC+8)
70031CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:23:41 (UTC+8)
70030CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:22:06 (UTC+8)
70029CUDA0: 14GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:20:31 (UTC+8)
70028CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:18:57 (UTC+8)
70027CUDA0: 14GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:17:22 (UTC+8)
70026CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:15:47 (UTC+8)
70025CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:14:12 (UTC+8)
70024CUDA0: 14GB / 48GB 92% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:12:37 (UTC+8)
70023CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:09:23 (UTC+8)
70022CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:07:49 (UTC+8)
70021CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 83% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:06:14 (UTC+8)
70020CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:04:39 (UTC+8)
70019CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:03:04 (UTC+8)
70018CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-21:01:29 (UTC+8)
70017CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:59:55 (UTC+8)
70016CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:58:20 (UTC+8)
70015CUDA0: 14GB / 48GB 94% || CUDA1: 8GB / 48GB 65% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:55:06 (UTC+8)
70014CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:53:31 (UTC+8)
70013CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:51:56 (UTC+8)
70012CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:50:21 (UTC+8)
70011CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:48:46 (UTC+8)
70010CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:47:11 (UTC+8)
70009CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:45:36 (UTC+8)
70008CUDA0: 14GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:44:02 (UTC+8)
70007CUDA0: 14GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:42:27 (UTC+8)
70006CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:40:52 (UTC+8)
70005CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:39:17 (UTC+8)
70004CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:37:42 (UTC+8)
70003CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:36:07 (UTC+8)
70002CUDA0: 14GB / 48GB 0% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:34:32 (UTC+8)
70001CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:32:58 (UTC+8)
70000CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:31:23 (UTC+8)
69999CUDA0: 14GB / 48GB 89% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:29:48 (UTC+8)
69998CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:28:14 (UTC+8)
69997CUDA0: 14GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:26:39 (UTC+8)
69996CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:25:04 (UTC+8)
69995CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 84% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:23:30 (UTC+8)
69994CUDA0: 14GB / 48GB 96% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:21:55 (UTC+8)
69993CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:20:20 (UTC+8)
69992CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:18:45 (UTC+8)
69991CUDA0: 14GB / 48GB 90% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:17:10 (UTC+8)
69990CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:15:36 (UTC+8)
69989CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:14:01 (UTC+8)
69988CUDA0: 14GB / 48GB 83% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:12:26 (UTC+8)
69987CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:10:51 (UTC+8)
69986CUDA0: 14GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:09:17 (UTC+8)
69985CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:07:43 (UTC+8)
69984CUDA0: 14GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:06:08 (UTC+8)
69983CUDA0: 14GB / 48GB 92% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:04:33 (UTC+8)
69982CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:02:58 (UTC+8)
69981CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-20:01:23 (UTC+8)
69980CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:59:48 (UTC+8)
69979CUDA0: 14GB / 48GB 94% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:58:14 (UTC+8)
69978CUDA0: 14GB / 48GB 93% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:56:39 (UTC+8)
69977CUDA0: 14GB / 48GB 94% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:55:05 (UTC+8)
69976CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:53:30 (UTC+8)
69975CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:51:55 (UTC+8)
69974CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 0% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:50:20 (UTC+8)
69973CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:48:45 (UTC+8)
69972CUDA0: 14GB / 48GB 90% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:47:10 (UTC+8)
69971CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:45:36 (UTC+8)
69970CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:44:01 (UTC+8)
69969CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:42:26 (UTC+8)
69968CUDA0: 14GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:40:51 (UTC+8)
69967CUDA0: 14GB / 48GB 90% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:39:16 (UTC+8)
69966CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:37:42 (UTC+8)
69965CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:36:07 (UTC+8)
69964CUDA0: 14GB / 48GB 94% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:34:32 (UTC+8)
69963CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:32:57 (UTC+8)
69962CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:31:23 (UTC+8)
69961CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:29:48 (UTC+8)
69960CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:28:13 (UTC+8)
69959CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:26:38 (UTC+8)
69958CUDA0: 14GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:25:03 (UTC+8)
69957CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 83% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:23:28 (UTC+8)
69956CUDA0: 14GB / 48GB 93% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:21:53 (UTC+8)
69955CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:20:19 (UTC+8)
69954CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:17:05 (UTC+8)
69953CUDA0: 14GB / 48GB 39% || CUDA1: 8GB / 48GB 58% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:15:30 (UTC+8)
69952CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:13:55 (UTC+8)
69951CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:12:21 (UTC+8)
69950CUDA0: 14GB / 48GB 89% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:10:46 (UTC+8)
69949CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:09:11 (UTC+8)
69948CUDA0: 14GB / 48GB 97% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:07:36 (UTC+8)
69947CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:06:01 (UTC+8)
69946CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:04:27 (UTC+8)
69945CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:02:52 (UTC+8)
69944CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-19:01:18 (UTC+8)
69943CUDA0: 14GB / 48GB 96% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:59:43 (UTC+8)
69942CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:58:08 (UTC+8)
69941CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:56:33 (UTC+8)
69940CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 82% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:54:59 (UTC+8)
69939CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:53:24 (UTC+8)
69938CUDA0: 14GB / 48GB 71% || CUDA1: 8GB / 48GB 0% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:51:49 (UTC+8)
69937CUDA0: 14GB / 48GB 97% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:50:14 (UTC+8)
69936CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:48:40 (UTC+8)
69935CUDA0: 14GB / 48GB 0% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:47:05 (UTC+8)
69934CUDA0: 14GB / 48GB 88% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:45:30 (UTC+8)
69933CUDA0: 14GB / 48GB 90% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:43:55 (UTC+8)
69932CUDA0: 14GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:42:21 (UTC+8)
69931CUDA0: 14GB / 48GB 91% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:40:46 (UTC+8)
69930CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 48% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:39:11 (UTC+8)
69929CUDA0: 14GB / 48GB 64% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:37:37 (UTC+8)
69928CUDA0: 14GB / 48GB 91% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:36:02 (UTC+8)
69927CUDA0: 14GB / 48GB 96% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:34:28 (UTC+8)
69926CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:32:53 (UTC+8)
69925CUDA0: 14GB / 48GB 92% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:29:39 (UTC+8)
69924CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:28:05 (UTC+8)
69923CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:26:30 (UTC+8)
69922CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:24:55 (UTC+8)
69921CUDA0: 14GB / 48GB 95% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:23:20 (UTC+8)
69920CUDA0: 14GB / 48GB 89% || CUDA1: 8GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:21:46 (UTC+8)
69919CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:20:11 (UTC+8)
69918CUDA0: 14GB / 48GB 100% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:18:36 (UTC+8)
69917CUDA0: 14GB / 48GB 91% || CUDA1: 8GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:17:01 (UTC+8)
69916CUDA0: 14GB / 48GB 32% || CUDA1: 8GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 7856MB)260819-18:12:09 (UTC+8)
69915CUDA0: 14GB / 48GB 90% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12610MB)260819-18:10:34 (UTC+8)
69914CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12610MB)260819-18:08:59 (UTC+8)
69913CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12610MB)260819-18:07:25 (UTC+8)
69912CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12610MB)260819-18:05:50 (UTC+8)
69911CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-18:04:15 (UTC+8)
69910CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-18:02:40 (UTC+8)
69909CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-18:01:05 (UTC+8)
69908CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-17:59:30 (UTC+8)
69907CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-17:57:55 (UTC+8)
69906CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-17:56:21 (UTC+8)
69905CUDA0: 14GB / 48GB 49% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-17:54:46 (UTC+8)
69904CUDA0: 14GB / 48GB 96% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-17:51:33 (UTC+8)
69903CUDA0: 14GB / 48GB 90% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-17:49:58 (UTC+8)
69902CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12610MB)260819-17:48:23 (UTC+8)
69901CUDA0: 14GB / 48GB 80% || CUDA1: 12GB / 48GB 70% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12598MB)260819-17:46:49 (UTC+8)
69900CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12350MB | CUDA1 PID:[2633805] 12598MB)260819-17:45:14 (UTC+8)
69899CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12348MB | CUDA1 PID:[2633805] 12654MB)260819-17:43:39 (UTC+8)
69898CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12348MB | CUDA1 PID:[2633805] 12654MB)260819-17:42:04 (UTC+8)
69897CUDA0: 14GB / 48GB 96% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12338MB | CUDA1 PID:[2633805] 12654MB)260819-17:40:29 (UTC+8)
69896CUDA0: 14GB / 48GB 74% || CUDA1: 12GB / 48GB 87% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12338MB | CUDA1 PID:[2633805] 12654MB)260819-17:38:55 (UTC+8)
69895CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:37:20 (UTC+8)
69894CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:35:45 (UTC+8)
69893CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:34:11 (UTC+8)
69892CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:32:36 (UTC+8)
69891CUDA0: 14GB / 48GB 96% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:31:02 (UTC+8)
69890CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:29:27 (UTC+8)
69889CUDA0: 14GB / 48GB 95% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:27:52 (UTC+8)
69888CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:26:18 (UTC+8)
69887CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:24:43 (UTC+8)
69886CUDA0: 14GB / 48GB 90% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:23:08 (UTC+8)
69885CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:21:33 (UTC+8)
69884CUDA0: 14GB / 48GB 80% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:19:59 (UTC+8)
69883CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:18:24 (UTC+8)
69882CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 69% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:16:49 (UTC+8)
69881CUDA0: 14GB / 48GB 96% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:15:15 (UTC+8)
69880CUDA0: 14GB / 48GB 93% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:13:40 (UTC+8)
69879CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:10:26 (UTC+8)
69878CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 55% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:08:52 (UTC+8)
69877CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:07:17 (UTC+8)
69876CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:05:42 (UTC+8)
69875CUDA0: 14GB / 48GB 97% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:04:07 (UTC+8)
69874CUDA0: 14GB / 48GB 93% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:02:33 (UTC+8)
69873CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-17:00:58 (UTC+8)
69872CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:59:23 (UTC+8)
69871CUDA0: 14GB / 48GB 90% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:57:48 (UTC+8)
69870CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:56:13 (UTC+8)
69869CUDA0: 14GB / 48GB 95% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:54:38 (UTC+8)
69868CUDA0: 14GB / 48GB 0% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:51:25 (UTC+8)
69867CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:49:50 (UTC+8)
69866CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:48:15 (UTC+8)
69865CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:46:40 (UTC+8)
69864CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:45:06 (UTC+8)
69863CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:43:31 (UTC+8)
69862CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:41:56 (UTC+8)
69861CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:40:21 (UTC+8)
69860CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12392MB | CUDA1 PID:[2633805] 12654MB)260819-16:38:46 (UTC+8)
69859CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:37:12 (UTC+8)
69858CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:35:37 (UTC+8)
69857CUDA0: 14GB / 48GB 98% || CUDA1: 12GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:34:02 (UTC+8)
69856CUDA0: 14GB / 48GB 99% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:32:28 (UTC+8)
69855CUDA0: 14GB / 48GB 90% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:30:53 (UTC+8)
69854CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:29:19 (UTC+8)
69853CUDA0: 14GB / 48GB 96% || CUDA1: 12GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:27:44 (UTC+8)
69852CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 82% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:26:09 (UTC+8)
69851CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:24:34 (UTC+8)
69850CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:22:59 (UTC+8)
69849CUDA0: 14GB / 48GB 87% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:21:25 (UTC+8)
69848CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:19:50 (UTC+8)
69847CUDA0: 14GB / 48GB 94% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:18:15 (UTC+8)
69846CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:16:40 (UTC+8)
69845CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:15:06 (UTC+8)
69844CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:13:31 (UTC+8)
69843CUDA0: 14GB / 48GB 94% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:11:56 (UTC+8)
69842CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:10:21 (UTC+8)
69841CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12394MB | CUDA1 PID:[2633805] 12654MB)260819-16:08:47 (UTC+8)
69840CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-16:07:12 (UTC+8)
69839CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-16:05:37 (UTC+8)
69838CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-16:04:02 (UTC+8)
69837CUDA0: 14GB / 48GB 95% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-16:02:28 (UTC+8)
69836CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-16:00:53 (UTC+8)
69835CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-15:59:18 (UTC+8)
69834CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-15:57:43 (UTC+8)
69833CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-15:56:09 (UTC+8)
69832CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-15:54:34 (UTC+8)
69831CUDA0: 14GB / 48GB 80% || CUDA1: 12GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-15:53:00 (UTC+8)
69830CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-15:51:25 (UTC+8)
69829CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-15:49:50 (UTC+8)
69828CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-15:46:36 (UTC+8)
69827CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12654MB)260819-15:45:02 (UTC+8)
69826CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12708MB)260819-15:43:27 (UTC+8)
69825CUDA0: 14GB / 48GB 73% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12708MB)260819-15:41:52 (UTC+8)
69824CUDA0: 14GB / 48GB 96% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12708MB)260819-15:40:17 (UTC+8)
69823CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12396MB | CUDA1 PID:[2633805] 12708MB)260819-15:38:42 (UTC+8)
69822CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:35:28 (UTC+8)
69821CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:33:54 (UTC+8)
69820CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:32:19 (UTC+8)
69819CUDA0: 14GB / 48GB 90% || CUDA1: 12GB / 48GB 0% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:30:44 (UTC+8)
69818CUDA0: 14GB / 48GB 92% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:29:09 (UTC+8)
69817CUDA0: 14GB / 48GB 85% || CUDA1: 12GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:25:55 (UTC+8)
69816CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:24:21 (UTC+8)
69815CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:22:46 (UTC+8)
69814CUDA0: 14GB / 48GB 90% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:21:11 (UTC+8)
69813CUDA0: 14GB / 48GB 91% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:19:36 (UTC+8)
69812CUDA0: 14GB / 48GB 94% || CUDA1: 12GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:18:02 (UTC+8)
69811CUDA0: 14GB / 48GB 11% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:16:27 (UTC+8)
69810CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12708MB)260819-15:14:53 (UTC+8)
69809CUDA0: 14GB / 48GB 97% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12968MB)260819-15:13:18 (UTC+8)
69808CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12968MB)260819-15:11:43 (UTC+8)
69807CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12968MB)260819-15:08:29 (UTC+8)
69806CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-15:06:54 (UTC+8)
69805CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-15:05:20 (UTC+8)
69804CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-15:03:45 (UTC+8)
69803CUDA0: 14GB / 48GB 90% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-15:02:10 (UTC+8)
69802CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-15:00:35 (UTC+8)
69801CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:59:01 (UTC+8)
69800CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:57:26 (UTC+8)
69799CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:55:51 (UTC+8)
69798CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:54:17 (UTC+8)
69797CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:52:42 (UTC+8)
69796CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:51:07 (UTC+8)
69795CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:49:32 (UTC+8)
69794CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:47:57 (UTC+8)
69793CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:46:22 (UTC+8)
69792CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:44:47 (UTC+8)
69791CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:43:13 (UTC+8)
69790CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:41:38 (UTC+8)
69789CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:40:03 (UTC+8)
69788CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:36:49 (UTC+8)
69787CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:35:14 (UTC+8)
69786CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:33:39 (UTC+8)
69785CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:32:04 (UTC+8)
69784CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:30:29 (UTC+8)
69783CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:28:54 (UTC+8)
69782CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:27:20 (UTC+8)
69781CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:25:45 (UTC+8)
69780CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:24:10 (UTC+8)
69779CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:22:36 (UTC+8)
69778CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:21:01 (UTC+8)
69777CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:19:26 (UTC+8)
69776CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:17:51 (UTC+8)
69775CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:16:16 (UTC+8)
69774CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:14:41 (UTC+8)
69773CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:13:07 (UTC+8)
69772CUDA0: 14GB / 48GB 98% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:11:32 (UTC+8)
69771CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:09:57 (UTC+8)
69770CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:08:22 (UTC+8)
69769CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:06:48 (UTC+8)
69768CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:05:13 (UTC+8)
69767CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:03:38 (UTC+8)
69766CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:02:04 (UTC+8)
69765CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-14:00:29 (UTC+8)
69764CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:58:54 (UTC+8)
69763CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:57:20 (UTC+8)
69762CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:55:45 (UTC+8)
69761CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 13% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:54:10 (UTC+8)
69760CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 98% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:52:35 (UTC+8)
69759CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:51:00 (UTC+8)
69758CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:49:26 (UTC+8)
69757CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:47:51 (UTC+8)
69756CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:46:16 (UTC+8)
69755CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:44:41 (UTC+8)
69754CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:43:07 (UTC+8)
69753CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:41:32 (UTC+8)
69752CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:39:57 (UTC+8)
69751CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:38:22 (UTC+8)
69750CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:36:47 (UTC+8)
69749CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:35:13 (UTC+8)
69748CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:33:38 (UTC+8)
69747CUDA0: 14GB / 48GB 38% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:32:03 (UTC+8)
69746CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:30:29 (UTC+8)
69745CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:28:54 (UTC+8)
69744CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:27:19 (UTC+8)
69743CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 89% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:25:44 (UTC+8)
69742CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:24:10 (UTC+8)
69741CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:22:35 (UTC+8)
69740CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:21:01 (UTC+8)
69739CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:19:26 (UTC+8)
69738CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:17:52 (UTC+8)
69737CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:16:17 (UTC+8)
69736CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:14:43 (UTC+8)
69735CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:13:08 (UTC+8)
69734CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:11:33 (UTC+8)
69733CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:09:59 (UTC+8)
69732CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:08:24 (UTC+8)
69731CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:06:50 (UTC+8)
69730CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-13:05:15 (UTC+8)
69729CUDA0: 14GB / 48GB 44% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-13:03:40 (UTC+8)
69728CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-13:02:05 (UTC+8)
69727CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-13:00:31 (UTC+8)
69726CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12446MB | CUDA1 PID:[2633805] 12970MB)260819-12:58:56 (UTC+8)
69725CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:57:21 (UTC+8)
69724CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:55:47 (UTC+8)
69723CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:54:12 (UTC+8)
69722CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:52:38 (UTC+8)
69721CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:51:03 (UTC+8)
69720CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:49:28 (UTC+8)
69719CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:47:54 (UTC+8)
69718CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:44:40 (UTC+8)
69717CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:43:05 (UTC+8)
69716CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:39:51 (UTC+8)
69715CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:38:17 (UTC+8)
69714CUDA0: 14GB / 48GB 99% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:36:42 (UTC+8)
69713CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:35:07 (UTC+8)
69712CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 99% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:33:32 (UTC+8)
69711CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:31:57 (UTC+8)
69710CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:30:22 (UTC+8)
69709CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:28:48 (UTC+8)
69708CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:27:13 (UTC+8)
69707CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:25:39 (UTC+8)
69706CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:24:04 (UTC+8)
69705CUDA0: 14GB / 48GB 89% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-12:22:29 (UTC+8)
69704CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:20:54 (UTC+8)
69703CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:19:19 (UTC+8)
69702CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:17:44 (UTC+8)
69701CUDA0: 14GB / 48GB 89% || CUDA1: 13GB / 48GB 88% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:16:09 (UTC+8)
69700CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:14:34 (UTC+8)
69699CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:13:00 (UTC+8)
69698CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:11:25 (UTC+8)
69697CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:09:50 (UTC+8)
69696CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:08:16 (UTC+8)
69695CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:06:41 (UTC+8)
69694CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:05:06 (UTC+8)
69693CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:03:32 (UTC+8)
69692CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:01:57 (UTC+8)
69691CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-12:00:22 (UTC+8)
69690CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:58:47 (UTC+8)
69689CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:57:13 (UTC+8)
69688CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:55:38 (UTC+8)
69687CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:54:03 (UTC+8)
69686CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:52:29 (UTC+8)
69685CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:50:54 (UTC+8)
69684CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:49:19 (UTC+8)
69683CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:47:45 (UTC+8)
69682CUDA0: 14GB / 48GB 88% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:46:10 (UTC+8)
69681CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:44:35 (UTC+8)
69680CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:43:00 (UTC+8)
69679CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:41:25 (UTC+8)
69678CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:39:51 (UTC+8)
69677CUDA0: 14GB / 48GB 100% || CUDA1: 12GB / 48GB 18% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12710MB)260819-11:38:16 (UTC+8)
69676CUDA0: 14GB / 48GB 0% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:36:42 (UTC+8)
69675CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:35:07 (UTC+8)
69674CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:33:32 (UTC+8)
69673CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:31:57 (UTC+8)
69672CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:30:22 (UTC+8)
69671CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:28:47 (UTC+8)
69670CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:27:12 (UTC+8)
69669CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:25:38 (UTC+8)
69668CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:24:03 (UTC+8)
69667CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:22:28 (UTC+8)
69666CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:20:53 (UTC+8)
69665CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:19:19 (UTC+8)
69664CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:17:44 (UTC+8)
69663CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:16:09 (UTC+8)
69662CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:14:34 (UTC+8)
69661CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:13:00 (UTC+8)
69660CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:09:46 (UTC+8)
69659CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:08:11 (UTC+8)
69658CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:06:36 (UTC+8)
69657CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 96% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:05:02 (UTC+8)
69656CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:03:27 (UTC+8)
69655CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:01:52 (UTC+8)
69654CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-11:00:17 (UTC+8)
69653CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-10:58:43 (UTC+8)
69652CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-10:57:08 (UTC+8)
69651CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-10:55:33 (UTC+8)
69650CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:52:19 (UTC+8)
69649CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:50:44 (UTC+8)
69648CUDA0: 14GB / 48GB 17% || CUDA1: 13GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12448MB | CUDA1 PID:[2633805] 12970MB)260819-10:49:09 (UTC+8)
69647CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:47:35 (UTC+8)
69646CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:46:00 (UTC+8)
69645CUDA0: 14GB / 48GB 90% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:44:25 (UTC+8)
69644CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:42:50 (UTC+8)
69643CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:41:15 (UTC+8)
69642CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:39:41 (UTC+8)
69641CUDA0: 14GB / 48GB 97% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:38:06 (UTC+8)
69640CUDA0: 14GB / 48GB 91% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:36:32 (UTC+8)
69639CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:34:57 (UTC+8)
69638CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:33:22 (UTC+8)
69637CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:31:47 (UTC+8)
69636CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:30:13 (UTC+8)
69635CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:28:38 (UTC+8)
69634CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:27:03 (UTC+8)
69633CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:25:29 (UTC+8)
69632CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:23:54 (UTC+8)
69631CUDA0: 14GB / 48GB 98% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:22:18 (UTC+8)
69630CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:20:44 (UTC+8)
69629CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:19:09 (UTC+8)
69628CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 11% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:17:35 (UTC+8)
69627CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:16:00 (UTC+8)
69626CUDA0: 14GB / 48GB 95% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:14:26 (UTC+8)
69625CUDA0: 14GB / 48GB 93% || CUDA1: 13GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12970MB)260819-10:12:51 (UTC+8)
69624CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-10:11:16 (UTC+8)
69623CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-10:09:41 (UTC+8)
69622CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 91% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-10:08:06 (UTC+8)
69621CUDA0: 14GB / 48GB 96% || CUDA1: 13GB / 48GB 94% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12706MB | CUDA1 PID:[2633805] 12968MB)260819-10:06:32 (UTC+8)
69620CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-10:04:57 (UTC+8)
69619CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 90% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-10:03:22 (UTC+8)
69618CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-10:01:48 (UTC+8)
69617CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-10:00:13 (UTC+8)
69616CUDA0: 14GB / 48GB 92% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:58:38 (UTC+8)
69615CUDA0: 14GB / 48GB 94% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:57:03 (UTC+8)
69614CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 92% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:55:28 (UTC+8)
69613CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:53:53 (UTC+8)
69612CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:52:19 (UTC+8)
69611CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 95% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:50:44 (UTC+8)
69610CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:49:09 (UTC+8)
69609CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:47:35 (UTC+8)
69608CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 93% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:46:00 (UTC+8)
69607CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:44:25 (UTC+8)
69606CUDA0: 14GB / 48GB 100% || CUDA1: 13GB / 48GB 100% [双卡负载] (高占用进程: CUDA0 PID:[2633803] 12708MB | CUDA1 PID:[2633805] 12968MB)260819-09:42:51 (UTC+8)
#04

Unknown_Full_EX

▦ table
last update
260822-00:30:59
records
641
type
text
latest
remaining=282; core_remaining=59; overflow_remaining=223; running=2; done=618; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-22T00:30:07+08:00
ID数值接收时间
641remaining=282; core_remaining=59; overflow_remaining=223; running=2; done=618; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-22T00:30:07+08:00260822-00:30:59 (UTC+8)
640remaining=282; core_remaining=59; overflow_remaining=223; running=2; done=618; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/93%; time=2026-08-22T00:20:05+08:00260822-00:20:32 (UTC+8)
639remaining=283; core_remaining=60; overflow_remaining=223; running=2; done=617; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-22T00:10:04+08:00260822-00:10:06 (UTC+8)
638remaining=284; core_remaining=61; overflow_remaining=223; running=2; done=616; failed=6; total=906; cuda0=23238MiB/91%; cuda1=23341MiB/100%; time=2026-08-21T23:59:38+08:00260821-23:59:39 (UTC+8)
637remaining=284; core_remaining=61; overflow_remaining=223; running=2; done=616; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T23:38:36+08:00260821-23:38:37 (UTC+8)
636remaining=284; core_remaining=61; overflow_remaining=223; running=2; done=616; failed=6; total=906; cuda0=23238MiB/90%; cuda1=23341MiB/100%; time=2026-08-21T23:28:08+08:00260821-23:28:11 (UTC+8)
635remaining=284; core_remaining=61; overflow_remaining=223; running=2; done=616; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T23:17:42+08:00260821-23:17:44 (UTC+8)
634remaining=284; core_remaining=61; overflow_remaining=223; running=2; done=616; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T23:07:16+08:00260821-23:07:18 (UTC+8)
633remaining=285; core_remaining=62; overflow_remaining=223; running=2; done=615; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T22:56:51+08:00260821-22:56:52 (UTC+8)
632remaining=286; core_remaining=63; overflow_remaining=223; running=2; done=614; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T22:46:25+08:00260821-22:46:25 (UTC+8)
631remaining=286; core_remaining=63; overflow_remaining=223; running=2; done=614; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T22:35:58+08:00260821-22:36:00 (UTC+8)
630remaining=286; core_remaining=63; overflow_remaining=223; running=2; done=614; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T22:25:31+08:00260821-22:25:33 (UTC+8)
629remaining=286; core_remaining=63; overflow_remaining=223; running=2; done=614; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T22:15:05+08:00260821-22:15:06 (UTC+8)
628remaining=286; core_remaining=63; overflow_remaining=223; running=2; done=614; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/88%; time=2026-08-21T22:04:39+08:00260821-22:04:40 (UTC+8)
627remaining=286; core_remaining=63; overflow_remaining=223; running=2; done=614; failed=6; total=906; cuda0=23238MiB/93%; cuda1=23341MiB/100%; time=2026-08-21T21:54:11+08:00260821-21:54:14 (UTC+8)
626remaining=286; core_remaining=63; overflow_remaining=223; running=2; done=614; failed=6; total=906; cuda0=23238MiB/89%; cuda1=23341MiB/100%; time=2026-08-21T21:43:45+08:00260821-21:43:47 (UTC+8)
625remaining=287; core_remaining=64; overflow_remaining=223; running=2; done=613; failed=6; total=906; cuda0=23088MiB/96%; cuda1=23341MiB/100%; time=2026-08-21T21:33:07+08:00260821-21:33:21 (UTC+8)
624remaining=288; core_remaining=65; overflow_remaining=223; running=2; done=612; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T21:22:53+08:00260821-21:22:54 (UTC+8)
623remaining=288; core_remaining=65; overflow_remaining=223; running=2; done=612; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T20:50:57+08:00260821-20:50:58 (UTC+8)
622remaining=288; core_remaining=65; overflow_remaining=223; running=2; done=612; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T20:40:30+08:00260821-20:40:32 (UTC+8)
621remaining=288; core_remaining=65; overflow_remaining=223; running=2; done=612; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T20:30:05+08:00260821-20:30:06 (UTC+8)
620remaining=289; core_remaining=66; overflow_remaining=223; running=2; done=611; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T20:19:38+08:00260821-20:19:40 (UTC+8)
619remaining=290; core_remaining=67; overflow_remaining=223; running=2; done=610; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T20:09:13+08:00260821-20:09:14 (UTC+8)
618remaining=290; core_remaining=67; overflow_remaining=223; running=2; done=610; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T19:58:47+08:00260821-19:58:48 (UTC+8)
617remaining=290; core_remaining=67; overflow_remaining=223; running=2; done=610; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/88%; time=2026-08-21T19:37:43+08:00260821-19:37:46 (UTC+8)
616remaining=290; core_remaining=67; overflow_remaining=223; running=2; done=610; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T19:27:19+08:00260821-19:27:20 (UTC+8)
615remaining=290; core_remaining=67; overflow_remaining=223; running=2; done=610; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/88%; time=2026-08-21T19:16:51+08:00260821-19:16:53 (UTC+8)
614remaining=292; core_remaining=69; overflow_remaining=223; running=2; done=608; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T18:55:50+08:00260821-18:55:51 (UTC+8)
613remaining=292; core_remaining=69; overflow_remaining=223; running=2; done=608; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T18:45:24+08:00260821-18:45:25 (UTC+8)
612remaining=292; core_remaining=69; overflow_remaining=223; running=2; done=608; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T18:34:58+08:00260821-18:34:59 (UTC+8)
611remaining=292; core_remaining=69; overflow_remaining=223; running=2; done=608; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T18:24:30+08:00260821-18:24:32 (UTC+8)
610remaining=292; core_remaining=69; overflow_remaining=223; running=2; done=608; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T18:14:05+08:00260821-18:14:06 (UTC+8)
609remaining=292; core_remaining=69; overflow_remaining=223; running=2; done=608; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T18:03:38+08:00260821-18:03:40 (UTC+8)
608remaining=294; core_remaining=71; overflow_remaining=223; running=2; done=606; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T17:41:23+08:00260821-17:42:38 (UTC+8)
607remaining=294; core_remaining=71; overflow_remaining=223; running=2; done=606; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/90%; time=2026-08-21T17:00:24+08:00260821-17:00:25 (UTC+8)
606remaining=294; core_remaining=71; overflow_remaining=223; running=2; done=606; failed=6; total=906; cuda0=23238MiB/52%; cuda1=23341MiB/100%; time=2026-08-21T16:49:58+08:00260821-16:49:59 (UTC+8)
605remaining=294; core_remaining=71; overflow_remaining=223; running=2; done=606; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23189MiB/100%; time=2026-08-21T16:38:53+08:00260821-16:39:33 (UTC+8)
604remaining=296; core_remaining=73; overflow_remaining=223; running=2; done=604; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/97%; time=2026-08-21T16:28:52+08:00260821-16:29:07 (UTC+8)
603remaining=296; core_remaining=73; overflow_remaining=223; running=2; done=604; failed=6; total=906; cuda0=23238MiB/89%; cuda1=23341MiB/89%; time=2026-08-21T16:18:40+08:00260821-16:18:41 (UTC+8)
602remaining=296; core_remaining=73; overflow_remaining=223; running=2; done=604; failed=6; total=906; cuda0=23238MiB/88%; cuda1=23341MiB/100%; time=2026-08-21T16:08:14+08:00260821-16:08:15 (UTC+8)
601remaining=296; core_remaining=73; overflow_remaining=223; running=2; done=604; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T15:57:48+08:00260821-15:57:49 (UTC+8)
600remaining=296; core_remaining=73; overflow_remaining=223; running=2; done=604; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T15:47:22+08:00260821-15:47:23 (UTC+8)
599remaining=296; core_remaining=73; overflow_remaining=223; running=2; done=604; failed=6; total=906; cuda0=23238MiB/95%; cuda1=23341MiB/100%; time=2026-08-21T15:36:55+08:00260821-15:36:57 (UTC+8)
598remaining=296; core_remaining=73; overflow_remaining=223; running=2; done=604; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T15:26:29+08:00260821-15:26:30 (UTC+8)
597remaining=297; core_remaining=74; overflow_remaining=223; running=2; done=603; failed=6; total=906; cuda0=23086MiB/20%; cuda1=23341MiB/100%; time=2026-08-21T15:16:02+08:00260821-15:16:04 (UTC+8)
596remaining=298; core_remaining=75; overflow_remaining=223; running=2; done=602; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T14:55:01+08:00260821-14:55:02 (UTC+8)
595remaining=298; core_remaining=75; overflow_remaining=223; running=2; done=602; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T14:23:24+08:00260821-14:23:25 (UTC+8)
594remaining=299; core_remaining=76; overflow_remaining=223; running=2; done=601; failed=6; total=906; cuda0=23238MiB/91%; cuda1=23341MiB/100%; time=2026-08-21T14:02:21+08:00260821-14:02:23 (UTC+8)
593remaining=300; core_remaining=77; overflow_remaining=223; running=2; done=600; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T12:58:41+08:00260821-12:58:42 (UTC+8)
592remaining=301; core_remaining=78; overflow_remaining=223; running=2; done=599; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/93%; time=2026-08-21T12:48:15+08:00260821-12:48:16 (UTC+8)
591remaining=302; core_remaining=79; overflow_remaining=223; running=2; done=598; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/94%; time=2026-08-21T12:26:22+08:00260821-12:27:14 (UTC+8)
590remaining=302; core_remaining=79; overflow_remaining=223; running=2; done=598; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T11:55:36+08:00260821-11:55:37 (UTC+8)
589remaining=304; core_remaining=81; overflow_remaining=223; running=2; done=596; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T11:13:12+08:00260821-11:13:14 (UTC+8)
588remaining=304; core_remaining=81; overflow_remaining=223; running=2; done=596; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/100%; time=2026-08-21T11:02:47+08:00260821-11:02:48 (UTC+8)
587remaining=304; core_remaining=81; overflow_remaining=223; running=2; done=596; failed=6; total=906; cuda0=23238MiB/100%; cuda1=23341MiB/94%; time=2026-08-21T10:41:21+08:00260821-10:41:46 (UTC+8)
586remaining=304; core_remaining=81; overflow_remaining=223; running=2; done=596; failed=6; total=906; cuda0=23238MiB/94%; cuda1=23341MiB/100%; time=2026-08-21T10:31:18+08:00260821-10:31:20 (UTC+8)
585remaining=306; core_remaining=83; overflow_remaining=223; running=2; done=594; failed=6; total=906; cuda0=23308MiB/100%; cuda1=21759MiB/100%; time=2026-08-21T10:09:56+08:00260821-10:10:18 (UTC+8)
584remaining=306; core_remaining=83; overflow_remaining=223; running=2; done=594; failed=6; total=906; cuda0=23308MiB/90%; cuda1=21759MiB/100%; time=2026-08-21T09:59:50+08:00260821-09:59:52 (UTC+8)
583remaining=306; core_remaining=83; overflow_remaining=223; running=2; done=594; failed=6; total=906; cuda0=23308MiB/100%; cuda1=21759MiB/100%; time=2026-08-21T09:49:22+08:00260821-09:49:26 (UTC+8)
582remaining=306; core_remaining=83; overflow_remaining=223; running=2; done=594; failed=6; total=906; cuda0=23308MiB/100%; cuda1=21759MiB/34%; time=2026-08-21T09:38:58+08:00260821-09:38:59 (UTC+8)
581remaining=306; core_remaining=83; overflow_remaining=223; running=2; done=594; failed=6; total=906; cuda0=23308MiB/100%; cuda1=21759MiB/100%; time=2026-08-21T09:28:31+08:00260821-09:28:33 (UTC+8)
580remaining=306; core_remaining=83; overflow_remaining=223; running=2; done=594; failed=6; total=906; cuda0=23308MiB/90%; cuda1=21759MiB/100%; time=2026-08-21T09:07:29+08:00260821-09:07:31 (UTC+8)
579remaining=308; core_remaining=85; overflow_remaining=223; running=2; done=592; failed=6; total=906; cuda0=14668MiB/100%; cuda1=13289MiB/100%; time=2026-08-21T08:57:03+08:00260821-08:57:04 (UTC+8)
578remaining=308; core_remaining=85; overflow_remaining=223; running=2; done=592; failed=6; total=906; cuda0=14668MiB/100%; cuda1=13289MiB/100%; time=2026-08-21T08:46:37+08:00260821-08:46:38 (UTC+8)
577remaining=308; core_remaining=85; overflow_remaining=223; running=2; done=592; failed=6; total=906; cuda0=14668MiB/100%; cuda1=13289MiB/100%; time=2026-08-21T08:35:18+08:00260821-08:36:12 (UTC+8)
576remaining=310; core_remaining=87; overflow_remaining=223; running=2; done=590; failed=6; total=906; cuda0=14670MiB/95%; cuda1=13287MiB/100%; time=2026-08-21T07:53:58+08:00260821-07:53:59 (UTC+8)
575remaining=310; core_remaining=87; overflow_remaining=223; running=2; done=590; failed=6; total=906; cuda0=14670MiB/89%; cuda1=13287MiB/96%; time=2026-08-21T07:43:32+08:00260821-07:43:33 (UTC+8)
574remaining=311; core_remaining=88; overflow_remaining=223; running=2; done=589; failed=6; total=906; cuda0=14670MiB/97%; cuda1=13291MiB/100%; time=2026-08-21T07:33:06+08:00260821-07:33:07 (UTC+8)
573remaining=312; core_remaining=89; overflow_remaining=223; running=2; done=588; failed=6; total=906; cuda0=14672MiB/100%; cuda1=13291MiB/91%; time=2026-08-21T07:22:39+08:00260821-07:22:41 (UTC+8)
572remaining=312; core_remaining=89; overflow_remaining=223; running=2; done=588; failed=6; total=906; cuda0=14672MiB/100%; cuda1=13291MiB/100%; time=2026-08-21T07:01:38+08:00260821-07:01:39 (UTC+8)
571remaining=314; core_remaining=91; overflow_remaining=223; running=2; done=586; failed=6; total=906; cuda0=14672MiB/100%; cuda1=13291MiB/94%; time=2026-08-21T06:30:01+08:00260821-06:30:02 (UTC+8)
570remaining=314; core_remaining=91; overflow_remaining=223; running=2; done=586; failed=6; total=906; cuda0=14672MiB/100%; cuda1=13291MiB/100%; time=2026-08-21T06:19:35+08:00260821-06:19:36 (UTC+8)
569remaining=316; core_remaining=93; overflow_remaining=223; running=2; done=584; failed=6; total=906; cuda0=14674MiB/100%; cuda1=13293MiB/100%; time=2026-08-21T05:58:12+08:00260821-05:58:34 (UTC+8)
568remaining=316; core_remaining=93; overflow_remaining=223; running=2; done=584; failed=6; total=906; cuda0=14674MiB/100%; cuda1=13293MiB/100%; time=2026-08-21T05:37:32+08:00260821-05:37:33 (UTC+8)
567remaining=318; core_remaining=95; overflow_remaining=223; running=2; done=582; failed=6; total=906; cuda0=14628MiB/100%; cuda1=12945MiB/100%; time=2026-08-21T04:44:43+08:00260821-04:44:44 (UTC+8)
566remaining=320; core_remaining=97; overflow_remaining=223; running=2; done=580; failed=6; total=906; cuda0=14068MiB/100%; cuda1=12383MiB/100%; time=2026-08-21T04:34:16+08:00260821-04:34:18 (UTC+8)
565remaining=320; core_remaining=97; overflow_remaining=223; running=2; done=580; failed=6; total=906; cuda0=14068MiB/100%; cuda1=12383MiB/100%; time=2026-08-21T04:23:51+08:00260821-04:23:52 (UTC+8)
564remaining=320; core_remaining=97; overflow_remaining=223; running=2; done=580; failed=6; total=906; cuda0=14068MiB/100%; cuda1=12383MiB/94%; time=2026-08-21T04:13:24+08:00260821-04:13:26 (UTC+8)
563remaining=322; core_remaining=99; overflow_remaining=223; running=2; done=578; failed=6; total=906; cuda0=14066MiB/95%; cuda1=12395MiB/100%; time=2026-08-21T03:41:38+08:00260821-03:41:38 (UTC+8)
562remaining=322; core_remaining=99; overflow_remaining=223; running=2; done=578; failed=6; total=906; cuda0=14066MiB/100%; cuda1=12395MiB/91%; time=2026-08-21T03:31:10+08:00260821-03:31:12 (UTC+8)
561remaining=323; core_remaining=100; overflow_remaining=223; running=2; done=577; failed=6; total=906; cuda0=14066MiB/100%; cuda1=12393MiB/100%; time=2026-08-21T03:20:45+08:00260821-03:20:46 (UTC+8)
560remaining=324; core_remaining=101; overflow_remaining=223; running=2; done=576; failed=6; total=906; cuda0=14070MiB/95%; cuda1=12393MiB/100%; time=2026-08-21T03:10:19+08:00260821-03:10:20 (UTC+8)
559remaining=324; core_remaining=101; overflow_remaining=223; running=2; done=576; failed=6; total=906; cuda0=14070MiB/100%; cuda1=12393MiB/100%; time=2026-08-21T02:59:52+08:00260821-02:59:54 (UTC+8)
558remaining=324; core_remaining=101; overflow_remaining=223; running=2; done=576; failed=6; total=906; cuda0=14070MiB/100%; cuda1=12393MiB/96%; time=2026-08-21T02:49:26+08:00260821-02:49:28 (UTC+8)
557remaining=326; core_remaining=103; overflow_remaining=223; running=2; done=574; failed=6; total=906; cuda0=14070MiB/93%; cuda1=12389MiB/100%; time=2026-08-21T02:28:23+08:00260821-02:28:26 (UTC+8)
556remaining=326; core_remaining=103; overflow_remaining=223; running=2; done=574; failed=6; total=906; cuda0=14070MiB/100%; cuda1=12389MiB/100%; time=2026-08-21T02:17:57+08:00260821-02:17:59 (UTC+8)
555remaining=326; core_remaining=103; overflow_remaining=223; running=2; done=574; failed=6; total=906; cuda0=14070MiB/93%; cuda1=12389MiB/100%; time=2026-08-21T02:07:32+08:00260821-02:07:33 (UTC+8)
554remaining=328; core_remaining=105; overflow_remaining=223; running=2; done=572; failed=6; total=906; cuda0=14068MiB/100%; cuda1=12397MiB/100%; time=2026-08-21T01:46:30+08:00260821-01:46:32 (UTC+8)
553remaining=328; core_remaining=105; overflow_remaining=223; running=2; done=572; failed=6; total=906; cuda0=14068MiB/93%; cuda1=12397MiB/95%; time=2026-08-21T01:36:04+08:00260821-01:36:05 (UTC+8)
552remaining=328; core_remaining=105; overflow_remaining=223; running=2; done=572; failed=6; total=906; cuda0=14068MiB/100%; cuda1=12397MiB/100%; time=2026-08-21T01:24:11+08:00260821-01:25:39 (UTC+8)
551remaining=330; core_remaining=107; overflow_remaining=223; running=2; done=570; failed=6; total=906; cuda0=13380MiB/100%; cuda1=11699MiB/100%; time=2026-08-21T01:14:10+08:00260821-01:15:13 (UTC+8)
550remaining=330; core_remaining=107; overflow_remaining=223; running=2; done=570; failed=6; total=906; cuda0=13380MiB/100%; cuda1=11699MiB/100%; time=2026-08-21T00:43:35+08:00260821-00:43:36 (UTC+8)
549remaining=332; core_remaining=109; overflow_remaining=223; running=2; done=568; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-21T00:33:08+08:00260821-00:33:10 (UTC+8)
548remaining=334; core_remaining=111; overflow_remaining=223; running=2; done=566; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7881MiB/100%; time=2026-08-21T00:12:06+08:00260821-00:12:08 (UTC+8)
547remaining=334; core_remaining=111; overflow_remaining=223; running=2; done=566; failed=6; total=906; cuda0=9518MiB/97%; cuda1=7881MiB/18%; time=2026-08-21T00:01:40+08:00260821-00:01:42 (UTC+8)
546remaining=336; core_remaining=113; overflow_remaining=223; running=2; done=564; failed=6; total=906; cuda0=9518MiB/90%; cuda1=7879MiB/94%; time=2026-08-20T23:40:37+08:00260820-23:40:39 (UTC+8)
545remaining=336; core_remaining=113; overflow_remaining=223; running=2; done=564; failed=6; total=906; cuda0=9518MiB/94%; cuda1=7879MiB/89%; time=2026-08-20T23:30:11+08:00260820-23:30:12 (UTC+8)
544remaining=337; core_remaining=114; overflow_remaining=223; running=2; done=563; failed=6; total=906; cuda0=9352MiB/100%; cuda1=7879MiB/99%; time=2026-08-20T23:19:43+08:00260820-23:19:46 (UTC+8)
543remaining=338; core_remaining=115; overflow_remaining=223; running=2; done=562; failed=6; total=906; cuda0=9518MiB/95%; cuda1=7879MiB/100%; time=2026-08-20T22:58:41+08:00260820-22:58:43 (UTC+8)
542remaining=340; core_remaining=117; overflow_remaining=223; running=2; done=560; failed=6; total=906; cuda0=9518MiB/95%; cuda1=7879MiB/100%; time=2026-08-20T22:48:15+08:00260820-22:48:16 (UTC+8)
541remaining=340; core_remaining=117; overflow_remaining=223; running=2; done=560; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/90%; time=2026-08-20T22:37:48+08:00260820-22:37:50 (UTC+8)
540remaining=341; core_remaining=118; overflow_remaining=223; running=2; done=559; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T22:27:22+08:00260820-22:27:24 (UTC+8)
539remaining=342; core_remaining=119; overflow_remaining=223; running=2; done=558; failed=6; total=906; cuda0=9518MiB/89%; cuda1=7879MiB/100%; time=2026-08-20T22:16:56+08:00260820-22:16:57 (UTC+8)
538remaining=342; core_remaining=119; overflow_remaining=223; running=2; done=558; failed=6; total=906; cuda0=9518MiB/89%; cuda1=7879MiB/90%; time=2026-08-20T22:06:30+08:00260820-22:06:31 (UTC+8)
537remaining=344; core_remaining=121; overflow_remaining=223; running=2; done=556; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7881MiB/100%; time=2026-08-20T21:45:27+08:00260820-21:45:29 (UTC+8)
536remaining=345; core_remaining=122; overflow_remaining=223; running=2; done=555; failed=6; total=906; cuda0=9518MiB/89%; cuda1=7879MiB/92%; time=2026-08-20T21:35:00+08:00260820-21:35:03 (UTC+8)
535remaining=346; core_remaining=123; overflow_remaining=223; running=2; done=554; failed=6; total=906; cuda0=9518MiB/91%; cuda1=7879MiB/100%; time=2026-08-20T21:23:57+08:00260820-21:24:35 (UTC+8)
534remaining=346; core_remaining=123; overflow_remaining=223; running=2; done=554; failed=6; total=906; cuda0=9518MiB/81%; cuda1=7879MiB/90%; time=2026-08-20T21:13:56+08:00260820-21:14:09 (UTC+8)
533remaining=348; core_remaining=125; overflow_remaining=223; running=2; done=552; failed=6; total=906; cuda0=9520MiB/100%; cuda1=7881MiB/91%; time=2026-08-20T21:03:42+08:00260820-21:03:43 (UTC+8)
532remaining=348; core_remaining=125; overflow_remaining=223; running=2; done=552; failed=6; total=906; cuda0=9520MiB/90%; cuda1=7881MiB/100%; time=2026-08-20T20:42:41+08:00260820-20:42:42 (UTC+8)
531remaining=350; core_remaining=127; overflow_remaining=223; running=2; done=550; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T20:21:39+08:00260820-20:21:40 (UTC+8)
530remaining=352; core_remaining=129; overflow_remaining=223; running=2; done=548; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T20:11:13+08:00260820-20:11:14 (UTC+8)
529remaining=352; core_remaining=129; overflow_remaining=223; running=2; done=548; failed=6; total=906; cuda0=9518MiB/91%; cuda1=7879MiB/95%; time=2026-08-20T20:00:47+08:00260820-20:00:48 (UTC+8)
528remaining=352; core_remaining=129; overflow_remaining=223; running=2; done=548; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/89%; time=2026-08-20T19:50:20+08:00260820-19:50:22 (UTC+8)
527remaining=354; core_remaining=131; overflow_remaining=223; running=2; done=546; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7881MiB/100%; time=2026-08-20T19:39:54+08:00260820-19:39:55 (UTC+8)
526remaining=356; core_remaining=133; overflow_remaining=223; running=2; done=544; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7881MiB/100%; time=2026-08-20T19:07:24+08:00260820-19:08:18 (UTC+8)
525remaining=356; core_remaining=133; overflow_remaining=223; running=2; done=544; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7881MiB/100%; time=2026-08-20T18:57:22+08:00260820-18:57:52 (UTC+8)
524remaining=358; core_remaining=135; overflow_remaining=223; running=2; done=542; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T18:47:21+08:00260820-18:47:25 (UTC+8)
523remaining=358; core_remaining=135; overflow_remaining=223; running=2; done=542; failed=6; total=906; cuda0=9518MiB/88%; cuda1=7879MiB/100%; time=2026-08-20T18:36:58+08:00260820-18:37:00 (UTC+8)
522remaining=359; core_remaining=136; overflow_remaining=223; running=2; done=541; failed=6; total=906; cuda0=9518MiB/89%; cuda1=7879MiB/100%; time=2026-08-20T18:26:32+08:00260820-18:26:33 (UTC+8)
521remaining=360; core_remaining=137; overflow_remaining=223; running=2; done=540; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T18:16:04+08:00260820-18:16:07 (UTC+8)
520remaining=360; core_remaining=137; overflow_remaining=223; running=2; done=540; failed=6; total=906; cuda0=9518MiB/77%; cuda1=7879MiB/53%; time=2026-08-20T18:05:39+08:00260820-18:05:41 (UTC+8)
519remaining=362; core_remaining=139; overflow_remaining=223; running=2; done=538; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/89%; time=2026-08-20T17:55:13+08:00260820-17:55:15 (UTC+8)
518remaining=362; core_remaining=139; overflow_remaining=223; running=2; done=538; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T17:44:47+08:00260820-17:44:49 (UTC+8)
517remaining=362; core_remaining=139; overflow_remaining=223; running=2; done=538; failed=6; total=906; cuda0=9518MiB/94%; cuda1=7879MiB/100%; time=2026-08-20T17:34:21+08:00260820-17:34:22 (UTC+8)
516remaining=364; core_remaining=141; overflow_remaining=223; running=2; done=536; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/90%; time=2026-08-20T17:23:55+08:00260820-17:23:57 (UTC+8)
515remaining=364; core_remaining=141; overflow_remaining=223; running=2; done=536; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T17:13:29+08:00260820-17:13:30 (UTC+8)
514remaining=364; core_remaining=141; overflow_remaining=223; running=2; done=536; failed=6; total=906; cuda0=9518MiB/91%; cuda1=7879MiB/100%; time=2026-08-20T17:03:02+08:00260820-17:03:04 (UTC+8)
513remaining=366; core_remaining=143; overflow_remaining=223; running=2; done=534; failed=6; total=906; cuda0=9518MiB/91%; cuda1=7881MiB/100%; time=2026-08-20T16:52:36+08:00260820-16:52:38 (UTC+8)
512remaining=366; core_remaining=143; overflow_remaining=223; running=2; done=534; failed=6; total=906; cuda0=9518MiB/96%; cuda1=7881MiB/100%; time=2026-08-20T16:42:11+08:00260820-16:42:12 (UTC+8)
511remaining=366; core_remaining=143; overflow_remaining=223; running=2; done=534; failed=6; total=906; cuda0=9518MiB/90%; cuda1=7881MiB/100%; time=2026-08-20T16:31:43+08:00260820-16:31:45 (UTC+8)
510remaining=368; core_remaining=145; overflow_remaining=223; running=2; done=532; failed=6; total=906; cuda0=9518MiB/89%; cuda1=7881MiB/100%; time=2026-08-20T16:21:18+08:00260820-16:21:19 (UTC+8)
509remaining=368; core_remaining=145; overflow_remaining=223; running=2; done=532; failed=6; total=906; cuda0=9518MiB/96%; cuda1=7881MiB/92%; time=2026-08-20T16:10:50+08:00260820-16:10:53 (UTC+8)
508remaining=368; core_remaining=145; overflow_remaining=223; running=2; done=532; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7881MiB/100%; time=2026-08-20T16:00:25+08:00260820-16:00:26 (UTC+8)
507remaining=370; core_remaining=147; overflow_remaining=223; running=2; done=530; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T15:49:58+08:00260820-15:50:00 (UTC+8)
506remaining=370; core_remaining=147; overflow_remaining=223; running=2; done=530; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T15:39:33+08:00260820-15:39:34 (UTC+8)
505remaining=371; core_remaining=148; overflow_remaining=223; running=2; done=529; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T15:29:06+08:00260820-15:29:08 (UTC+8)
504remaining=372; core_remaining=149; overflow_remaining=223; running=2; done=528; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T15:18:40+08:00260820-15:18:42 (UTC+8)
503remaining=372; core_remaining=149; overflow_remaining=223; running=2; done=528; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T15:08:14+08:00260820-15:08:16 (UTC+8)
502remaining=374; core_remaining=151; overflow_remaining=223; running=2; done=526; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7881MiB/91%; time=2026-08-20T14:47:12+08:00260820-14:47:14 (UTC+8)
501remaining=374; core_remaining=151; overflow_remaining=223; running=2; done=526; failed=6; total=906; cuda0=9518MiB/95%; cuda1=7881MiB/91%; time=2026-08-20T14:36:46+08:00260820-14:36:48 (UTC+8)
500remaining=376; core_remaining=153; overflow_remaining=223; running=2; done=524; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T14:26:20+08:00260820-14:26:21 (UTC+8)
499remaining=376; core_remaining=153; overflow_remaining=223; running=2; done=524; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T14:15:54+08:00260820-14:15:55 (UTC+8)
498remaining=376; core_remaining=153; overflow_remaining=223; running=2; done=524; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/96%; time=2026-08-20T14:05:26+08:00260820-14:05:29 (UTC+8)
497remaining=378; core_remaining=155; overflow_remaining=223; running=2; done=522; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/88%; time=2026-08-20T13:55:02+08:00260820-13:55:03 (UTC+8)
496remaining=378; core_remaining=155; overflow_remaining=223; running=2; done=522; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/95%; time=2026-08-20T13:44:35+08:00260820-13:44:37 (UTC+8)
495remaining=378; core_remaining=155; overflow_remaining=223; running=2; done=522; failed=6; total=906; cuda0=9518MiB/0%; cuda1=7879MiB/96%; time=2026-08-20T13:34:09+08:00260820-13:34:10 (UTC+8)
494remaining=380; core_remaining=157; overflow_remaining=223; running=2; done=520; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T13:23:42+08:00260820-13:23:44 (UTC+8)
493remaining=380; core_remaining=157; overflow_remaining=223; running=2; done=520; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/92%; time=2026-08-20T13:13:15+08:00260820-13:13:18 (UTC+8)
492remaining=380; core_remaining=157; overflow_remaining=223; running=2; done=520; failed=6; total=906; cuda0=9518MiB/96%; cuda1=7879MiB/100%; time=2026-08-20T13:02:50+08:00260820-13:02:51 (UTC+8)
491remaining=382; core_remaining=159; overflow_remaining=223; running=2; done=518; failed=6; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T12:51:45+08:00260820-12:52:25 (UTC+8)
490remaining=382; core_remaining=159; overflow_remaining=223; running=2; done=518; failed=6; total=906; cuda0=9518MiB/82%; cuda1=7879MiB/95%; time=2026-08-20T12:41:43+08:00260820-12:41:59 (UTC+8)
489remaining=382; core_remaining=159; overflow_remaining=223; running=2; done=518; failed=6; total=906; cuda0=9518MiB/90%; cuda1=7879MiB/100%; time=2026-08-20T12:31:29+08:00260820-12:31:32 (UTC+8)
488remaining=384; core_remaining=161; overflow_remaining=223; running=2; done=516; failed=6; total=906; cuda0=9518MiB/91%; cuda1=7881MiB/100%; time=2026-08-20T12:21:05+08:00260820-12:21:06 (UTC+8)
487remaining=384; core_remaining=161; overflow_remaining=223; running=2; done=516; failed=6; total=906; cuda0=9518MiB/95%; cuda1=7881MiB/100%; time=2026-08-20T12:10:38+08:00260820-12:10:40 (UTC+8)
486remaining=384; core_remaining=161; overflow_remaining=223; running=2; done=516; failed=6; total=906; cuda0=9518MiB/93%; cuda1=7881MiB/95%; time=2026-08-20T12:00:12+08:00260820-12:00:14 (UTC+8)
485remaining=386; core_remaining=163; overflow_remaining=223; running=2; done=514; failed=6; total=906; cuda0=20144MiB/59%; cuda1=18467MiB/100%; time=2026-08-20T11:49:47+08:00260820-11:49:48 (UTC+8)
484remaining=386; core_remaining=163; overflow_remaining=223; running=2; done=514; failed=6; total=906; cuda0=20144MiB/58%; cuda1=18467MiB/76%; time=2026-08-20T11:38:31+08:00260820-11:39:22 (UTC+8)
483remaining=387; core_remaining=164; overflow_remaining=223; running=2; done=513; failed=6; total=906; cuda0=20144MiB/91%; cuda1=18469MiB/100%; time=2026-08-20T11:28:30+08:00260820-11:28:56 (UTC+8)
482remaining=388; core_remaining=165; overflow_remaining=223; running=2; done=512; failed=6; total=906; cuda0=20148MiB/92%; cuda1=18469MiB/96%; time=2026-08-20T11:18:29+08:00260820-11:18:30 (UTC+8)
481remaining=388; core_remaining=165; overflow_remaining=223; running=2; done=512; failed=6; total=906; cuda0=20148MiB/100%; cuda1=18469MiB/100%; time=2026-08-20T11:08:02+08:00260820-11:08:04 (UTC+8)
480remaining=390; core_remaining=167; overflow_remaining=223; running=2; done=510; failed=6; total=906; cuda0=20128MiB/100%; cuda1=18449MiB/100%; time=2026-08-20T10:57:36+08:00260820-10:57:37 (UTC+8)
479remaining=390; core_remaining=167; overflow_remaining=223; running=2; done=510; failed=6; total=906; cuda0=20128MiB/96%; cuda1=18449MiB/100%; time=2026-08-20T10:47:09+08:00260820-10:47:11 (UTC+8)
478remaining=390; core_remaining=167; overflow_remaining=223; running=2; done=510; failed=6; total=906; cuda0=20128MiB/95%; cuda1=18449MiB/95%; time=2026-08-20T10:36:43+08:00260820-10:36:45 (UTC+8)
477remaining=392; core_remaining=169; overflow_remaining=223; running=2; done=508; failed=6; total=906; cuda0=20118MiB/100%; cuda1=18437MiB/100%; time=2026-08-20T10:26:17+08:00260820-10:26:19 (UTC+8)
476remaining=392; core_remaining=169; overflow_remaining=223; running=2; done=508; failed=6; total=906; cuda0=20118MiB/100%; cuda1=18437MiB/100%; time=2026-08-20T10:15:51+08:00260820-10:15:52 (UTC+8)
475remaining=392; core_remaining=169; overflow_remaining=223; running=2; done=508; failed=6; total=906; cuda0=20118MiB/100%; cuda1=18437MiB/100%; time=2026-08-20T10:05:20+08:00260820-10:05:26 (UTC+8)
474remaining=392; core_remaining=169; overflow_remaining=223; running=2; done=508; failed=6; total=906; cuda0=20118MiB/100%; cuda1=18437MiB/100%; time=2026-08-20T09:54:58+08:00260820-09:54:59 (UTC+8)
473remaining=392; core_remaining=169; overflow_remaining=223; running=2; done=508; failed=6; total=906; cuda0=20118MiB/100%; cuda1=18437MiB/100%; time=2026-08-20T09:44:31+08:00260820-09:44:33 (UTC+8)
472remaining=392; core_remaining=169; overflow_remaining=223; running=2; done=508; failed=6; total=906; cuda0=20118MiB/100%; cuda1=18437MiB/100%; time=2026-08-20T09:34:05+08:00260820-09:34:07 (UTC+8)
471remaining=392; core_remaining=169; overflow_remaining=223; running=2; done=508; failed=6; total=906; cuda0=20118MiB/100%; cuda1=18437MiB/99%; time=2026-08-20T09:23:39+08:00260820-09:23:40 (UTC+8)
470remaining=392; core_remaining=169; overflow_remaining=223; running=1; done=508; failed=6; total=906; cuda0=20112MiB/100%; cuda1=7713MiB/0%; time=2026-08-20T09:13:12+08:00260820-09:13:14 (UTC+8)
469remaining=395; core_remaining=172; overflow_remaining=223; running=2; done=508; failed=3; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/90%; time=2026-08-20T09:02:47+08:00260820-09:02:48 (UTC+8)
468remaining=395; core_remaining=172; overflow_remaining=223; running=2; done=508; failed=3; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/95%; time=2026-08-20T08:52:21+08:00260820-08:52:22 (UTC+8)
467remaining=395; core_remaining=172; overflow_remaining=223; running=2; done=508; failed=3; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/96%; time=2026-08-20T08:41:55+08:00260820-08:41:56 (UTC+8)
466remaining=395; core_remaining=172; overflow_remaining=223; running=2; done=508; failed=3; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/96%; time=2026-08-20T08:31:28+08:00260820-08:31:30 (UTC+8)
465remaining=395; core_remaining=172; overflow_remaining=223; running=2; done=508; failed=3; total=906; cuda0=9518MiB/96%; cuda1=7879MiB/100%; time=2026-08-20T08:21:02+08:00260820-08:21:03 (UTC+8)
464remaining=395; core_remaining=172; overflow_remaining=223; running=2; done=508; failed=3; total=906; cuda0=9518MiB/90%; cuda1=7879MiB/100%; time=2026-08-20T08:10:36+08:00260820-08:10:37 (UTC+8)
463remaining=395; core_remaining=172; overflow_remaining=223; running=2; done=508; failed=3; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T08:00:09+08:00260820-08:00:11 (UTC+8)
462remaining=397; core_remaining=174; overflow_remaining=223; running=2; done=508; failed=1; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/94%; time=2026-08-20T07:49:42+08:00260820-07:49:44 (UTC+8)
461remaining=398; core_remaining=175; overflow_remaining=223; running=2; done=508; failed=0; total=906; cuda0=9518MiB/76%; cuda1=7879MiB/93%; time=2026-08-20T07:39:17+08:00260820-07:39:18 (UTC+8)
460remaining=398; core_remaining=175; overflow_remaining=223; running=2; done=508; failed=0; total=906; cuda0=9518MiB/91%; cuda1=7879MiB/100%; time=2026-08-20T07:28:51+08:00260820-07:28:52 (UTC+8)
459remaining=398; core_remaining=175; overflow_remaining=223; running=2; done=508; failed=0; total=906; cuda0=9518MiB/90%; cuda1=7879MiB/0%; time=2026-08-20T07:18:24+08:00260820-07:18:26 (UTC+8)
458remaining=398; core_remaining=175; overflow_remaining=223; running=2; done=508; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/92%; time=2026-08-20T07:07:59+08:00260820-07:07:59 (UTC+8)
457remaining=398; core_remaining=175; overflow_remaining=223; running=2; done=508; failed=0; total=906; cuda0=9518MiB/90%; cuda1=7879MiB/100%; time=2026-08-20T06:57:32+08:00260820-06:57:33 (UTC+8)
456remaining=398; core_remaining=175; overflow_remaining=223; running=2; done=508; failed=0; total=906; cuda0=9520MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T06:47:06+08:00260820-06:47:07 (UTC+8)
455remaining=399; core_remaining=176; overflow_remaining=223; running=2; done=507; failed=0; total=906; cuda0=9520MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T06:36:39+08:00260820-06:36:41 (UTC+8)
454remaining=400; core_remaining=177; overflow_remaining=223; running=2; done=506; failed=0; total=906; cuda0=9520MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T06:26:14+08:00260820-06:26:15 (UTC+8)
453remaining=400; core_remaining=177; overflow_remaining=223; running=2; done=506; failed=0; total=906; cuda0=9520MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T06:15:48+08:00260820-06:15:50 (UTC+8)
452remaining=400; core_remaining=177; overflow_remaining=223; running=2; done=506; failed=0; total=906; cuda0=9520MiB/91%; cuda1=7879MiB/95%; time=2026-08-20T06:05:22+08:00260820-06:05:23 (UTC+8)
451remaining=401; core_remaining=178; overflow_remaining=223; running=2; done=505; failed=0; total=906; cuda0=9520MiB/96%; cuda1=7879MiB/100%; time=2026-08-20T05:54:55+08:00260820-05:54:57 (UTC+8)
450remaining=402; core_remaining=179; overflow_remaining=223; running=2; done=504; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T05:44:29+08:00260820-05:44:31 (UTC+8)
449remaining=402; core_remaining=179; overflow_remaining=223; running=2; done=504; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T05:23:27+08:00260820-05:23:29 (UTC+8)
448remaining=403; core_remaining=180; overflow_remaining=223; running=2; done=503; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T05:13:02+08:00260820-05:13:03 (UTC+8)
447remaining=404; core_remaining=181; overflow_remaining=223; running=2; done=502; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T05:02:36+08:00260820-05:02:37 (UTC+8)
446remaining=404; core_remaining=181; overflow_remaining=223; running=2; done=502; failed=0; total=906; cuda0=9518MiB/93%; cuda1=7879MiB/100%; time=2026-08-20T04:52:10+08:00260820-04:52:11 (UTC+8)
445remaining=404; core_remaining=181; overflow_remaining=223; running=2; done=502; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T04:41:44+08:00260820-04:41:45 (UTC+8)
444remaining=405; core_remaining=182; overflow_remaining=223; running=2; done=501; failed=0; total=906; cuda0=9352MiB/91%; cuda1=7879MiB/96%; time=2026-08-20T04:31:18+08:00260820-04:31:19 (UTC+8)
443remaining=406; core_remaining=183; overflow_remaining=223; running=2; done=500; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/96%; time=2026-08-20T04:20:52+08:00260820-04:20:53 (UTC+8)
442remaining=406; core_remaining=183; overflow_remaining=223; running=2; done=500; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T04:10:26+08:00260820-04:10:27 (UTC+8)
441remaining=408; core_remaining=185; overflow_remaining=223; running=2; done=498; failed=0; total=906; cuda0=9518MiB/95%; cuda1=7879MiB/93%; time=2026-08-20T04:00:00+08:00260820-04:00:01 (UTC+8)
440remaining=408; core_remaining=185; overflow_remaining=223; running=2; done=498; failed=0; total=906; cuda0=9518MiB/92%; cuda1=7879MiB/90%; time=2026-08-20T03:49:32+08:00260820-03:49:35 (UTC+8)
439remaining=409; core_remaining=186; overflow_remaining=223; running=2; done=497; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T03:39:08+08:00260820-03:39:09 (UTC+8)
438remaining=410; core_remaining=187; overflow_remaining=223; running=2; done=496; failed=0; total=906; cuda0=9518MiB/93%; cuda1=7879MiB/100%; time=2026-08-20T03:28:42+08:00260820-03:28:43 (UTC+8)
437remaining=410; core_remaining=187; overflow_remaining=223; running=2; done=496; failed=0; total=906; cuda0=9518MiB/93%; cuda1=7879MiB/100%; time=2026-08-20T03:18:15+08:00260820-03:18:17 (UTC+8)
436remaining=411; core_remaining=188; overflow_remaining=223; running=2; done=495; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/93%; time=2026-08-20T03:07:48+08:00260820-03:07:51 (UTC+8)
435remaining=412; core_remaining=189; overflow_remaining=223; running=2; done=494; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/93%; time=2026-08-20T02:56:47+08:00260820-02:57:25 (UTC+8)
434remaining=412; core_remaining=189; overflow_remaining=223; running=2; done=494; failed=0; total=906; cuda0=9518MiB/90%; cuda1=7879MiB/90%; time=2026-08-20T02:46:45+08:00260820-02:46:59 (UTC+8)
433remaining=413; core_remaining=190; overflow_remaining=223; running=2; done=493; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T02:36:30+08:00260820-02:36:32 (UTC+8)
432remaining=414; core_remaining=191; overflow_remaining=223; running=2; done=492; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T02:26:05+08:00260820-02:26:06 (UTC+8)
431remaining=414; core_remaining=191; overflow_remaining=223; running=2; done=492; failed=0; total=906; cuda0=9518MiB/92%; cuda1=7879MiB/100%; time=2026-08-20T02:15:39+08:00260820-02:15:40 (UTC+8)
430remaining=415; core_remaining=192; overflow_remaining=223; running=2; done=491; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T02:05:13+08:00260820-02:05:14 (UTC+8)
429remaining=416; core_remaining=193; overflow_remaining=223; running=2; done=490; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T01:54:48+08:00260820-01:54:48 (UTC+8)
428remaining=416; core_remaining=193; overflow_remaining=223; running=2; done=490; failed=0; total=906; cuda0=9518MiB/95%; cuda1=7879MiB/96%; time=2026-08-20T01:44:22+08:00260820-01:44:23 (UTC+8)
427remaining=417; core_remaining=194; overflow_remaining=223; running=2; done=489; failed=0; total=906; cuda0=9518MiB/93%; cuda1=7879MiB/96%; time=2026-08-20T01:33:54+08:00260820-01:33:56 (UTC+8)
426remaining=418; core_remaining=195; overflow_remaining=223; running=2; done=488; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/98%; time=2026-08-20T01:23:28+08:00260820-01:23:30 (UTC+8)
425remaining=418; core_remaining=195; overflow_remaining=223; running=2; done=488; failed=0; total=906; cuda0=9518MiB/90%; cuda1=7879MiB/100%; time=2026-08-20T01:12:04+08:00260820-01:13:04 (UTC+8)
424remaining=420; core_remaining=197; overflow_remaining=223; running=2; done=486; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T01:02:03+08:00260820-01:02:37 (UTC+8)
423remaining=420; core_remaining=197; overflow_remaining=223; running=2; done=486; failed=0; total=906; cuda0=9518MiB/95%; cuda1=7879MiB/90%; time=2026-08-20T00:52:02+08:00260820-00:52:11 (UTC+8)
422remaining=421; core_remaining=198; overflow_remaining=223; running=2; done=485; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/91%; time=2026-08-20T00:41:43+08:00260820-00:41:45 (UTC+8)
421remaining=422; core_remaining=199; overflow_remaining=223; running=2; done=484; failed=0; total=906; cuda0=9518MiB/96%; cuda1=7879MiB/97%; time=2026-08-20T00:31:17+08:00260820-00:31:19 (UTC+8)
420remaining=422; core_remaining=199; overflow_remaining=223; running=2; done=484; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-20T00:20:51+08:00260820-00:20:53 (UTC+8)
419remaining=424; core_remaining=201; overflow_remaining=223; running=2; done=482; failed=0; total=906; cuda0=9518MiB/94%; cuda1=7879MiB/92%; time=2026-08-20T00:10:25+08:00260820-00:10:26 (UTC+8)
418remaining=424; core_remaining=201; overflow_remaining=223; running=2; done=482; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-19T23:59:59+08:00260820-00:00:00 (UTC+8)
417remaining=425; core_remaining=202; overflow_remaining=223; running=2; done=481; failed=0; total=906; cuda0=9518MiB/76%; cuda1=7879MiB/100%; time=2026-08-19T23:49:31+08:00260819-23:49:34 (UTC+8)
416remaining=426; core_remaining=203; overflow_remaining=223; running=2; done=480; failed=0; total=906; cuda0=9518MiB/14%; cuda1=7879MiB/100%; time=2026-08-19T23:39:05+08:00260819-23:39:07 (UTC+8)
415remaining=426; core_remaining=203; overflow_remaining=223; running=2; done=480; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-19T23:28:38+08:00260819-23:28:40 (UTC+8)
414remaining=427; core_remaining=204; overflow_remaining=223; running=2; done=479; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-19T23:18:12+08:00260819-23:18:14 (UTC+8)
413remaining=428; core_remaining=205; overflow_remaining=223; running=2; done=478; failed=0; total=906; cuda0=9520MiB/100%; cuda1=7879MiB/88%; time=2026-08-19T23:07:46+08:00260819-23:07:47 (UTC+8)
412remaining=429; core_remaining=206; overflow_remaining=223; running=2; done=477; failed=0; total=906; cuda0=9520MiB/100%; cuda1=7879MiB/100%; time=2026-08-19T22:57:19+08:00260819-22:57:20 (UTC+8)
411remaining=430; core_remaining=207; overflow_remaining=223; running=2; done=476; failed=0; total=906; cuda0=9518MiB/90%; cuda1=7879MiB/100%; time=2026-08-19T22:46:53+08:00260819-22:46:54 (UTC+8)
410remaining=430; core_remaining=207; overflow_remaining=223; running=2; done=476; failed=0; total=906; cuda0=9518MiB/100%; cuda1=7879MiB/100%; time=2026-08-19T22:36:26+08:00260819-22:36:28 (UTC+8)
409remaining=431; core_remaining=208; overflow_remaining=223; running=2; done=475; failed=0; total=906; cuda0=9518MiB/7%; cuda1=7879MiB/100%; time=2026-08-19T22:26:00+08:00260819-22:26:02 (UTC+8)
408remaining=432; core_remaining=209; overflow_remaining=223; running=2; done=474; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/100%; time=2026-08-19T22:15:34+08:00260819-22:15:36 (UTC+8)
407remaining=432; core_remaining=209; overflow_remaining=223; running=2; done=474; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/95%; time=2026-08-19T22:05:08+08:00260819-22:05:10 (UTC+8)
406remaining=433; core_remaining=210; overflow_remaining=223; running=2; done=473; failed=0; total=906; cuda0=14134MiB/91%; cuda1=7879MiB/91%; time=2026-08-19T21:54:42+08:00260819-21:54:43 (UTC+8)
405remaining=434; core_remaining=211; overflow_remaining=223; running=2; done=472; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/91%; time=2026-08-19T21:44:16+08:00260819-21:44:17 (UTC+8)
404remaining=434; core_remaining=211; overflow_remaining=223; running=2; done=472; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/89%; time=2026-08-19T21:33:49+08:00260819-21:33:51 (UTC+8)
403remaining=435; core_remaining=212; overflow_remaining=223; running=2; done=471; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/95%; time=2026-08-19T21:23:23+08:00260819-21:23:25 (UTC+8)
402remaining=436; core_remaining=213; overflow_remaining=223; running=2; done=470; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/96%; time=2026-08-19T21:12:57+08:00260819-21:12:58 (UTC+8)
401remaining=436; core_remaining=213; overflow_remaining=223; running=2; done=470; failed=0; total=906; cuda0=14134MiB/92%; cuda1=7879MiB/93%; time=2026-08-19T21:02:30+08:00260819-21:02:32 (UTC+8)
400remaining=437; core_remaining=214; overflow_remaining=223; running=2; done=469; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/0%; time=2026-08-19T20:51:15+08:00260819-20:52:05 (UTC+8)
399remaining=438; core_remaining=215; overflow_remaining=223; running=2; done=468; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/96%; time=2026-08-19T20:41:14+08:00260819-20:41:39 (UTC+8)
398remaining=438; core_remaining=215; overflow_remaining=223; running=2; done=468; failed=0; total=906; cuda0=14134MiB/34%; cuda1=7879MiB/100%; time=2026-08-19T20:31:11+08:00260819-20:31:13 (UTC+8)
397remaining=440; core_remaining=217; overflow_remaining=223; running=2; done=466; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/100%; time=2026-08-19T20:10:10+08:00260819-20:10:11 (UTC+8)
396remaining=440; core_remaining=217; overflow_remaining=223; running=2; done=466; failed=0; total=906; cuda0=14134MiB/93%; cuda1=7879MiB/100%; time=2026-08-19T19:59:43+08:00260819-19:59:45 (UTC+8)
395remaining=442; core_remaining=219; overflow_remaining=223; running=2; done=464; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/100%; time=2026-08-19T19:49:19+08:00260819-19:49:19 (UTC+8)
394remaining=442; core_remaining=219; overflow_remaining=223; running=2; done=464; failed=0; total=906; cuda0=14134MiB/90%; cuda1=7879MiB/90%; time=2026-08-19T19:38:51+08:00260819-19:38:53 (UTC+8)
393remaining=443; core_remaining=220; overflow_remaining=223; running=2; done=463; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/100%; time=2026-08-19T19:28:26+08:00260819-19:28:27 (UTC+8)
392remaining=444; core_remaining=221; overflow_remaining=223; running=2; done=462; failed=0; total=906; cuda0=14134MiB/91%; cuda1=7879MiB/96%; time=2026-08-19T19:18:00+08:00260819-19:18:01 (UTC+8)
391remaining=444; core_remaining=221; overflow_remaining=223; running=2; done=462; failed=0; total=906; cuda0=14134MiB/89%; cuda1=7879MiB/56%; time=2026-08-19T19:07:34+08:00260819-19:07:35 (UTC+8)
390remaining=446; core_remaining=223; overflow_remaining=223; running=2; done=460; failed=0; total=906; cuda0=14134MiB/100%; cuda1=7879MiB/55%; time=2026-08-19T18:57:07+08:00260819-18:57:09 (UTC+8)
389remaining=446; core_remaining=223; overflow_remaining=223; running=2; done=460; failed=0; total=906; cuda0=14134MiB/94%; cuda1=7879MiB/71%; time=2026-08-19T18:46:39+08:00260819-18:46:43 (UTC+8)
388remaining=448; core_remaining=225; overflow_remaining=223; running=2; done=458; failed=0; total=906; cuda0=14134MiB/98%; cuda1=7879MiB/100%; time=2026-08-19T18:25:40+08:00260819-18:25:41 (UTC+8)
387remaining=448; core_remaining=225; overflow_remaining=223; running=2; done=458; failed=0; total=906; cuda0=14134MiB/64%; cuda1=7879MiB/100%; time=2026-08-19T18:15:14+08:00260819-18:15:15 (UTC+8)
386remaining=449; core_remaining=226; overflow_remaining=223; running=2; done=457; failed=0; total=906; cuda0=14134MiB/91%; cuda1=12633MiB/100%; time=2026-08-19T18:04:48+08:00260819-18:04:49 (UTC+8)
385remaining=450; core_remaining=227; overflow_remaining=223; running=2; done=456; failed=0; total=906; cuda0=14092MiB/94%; cuda1=12633MiB/100%; time=2026-08-19T17:54:21+08:00260819-17:54:23 (UTC+8)
384remaining=452; core_remaining=229; overflow_remaining=223; running=2; done=454; failed=0; total=906; cuda0=14134MiB/87%; cuda1=12677MiB/100%; time=2026-08-19T17:33:20+08:00260819-17:33:21 (UTC+8)
383remaining=452; core_remaining=229; overflow_remaining=223; running=2; done=454; failed=0; total=906; cuda0=14134MiB/100%; cuda1=12677MiB/100%; time=2026-08-19T17:22:54+08:00260819-17:22:55 (UTC+8)
382remaining=453; core_remaining=230; overflow_remaining=223; running=2; done=453; failed=0; total=906; cuda0=14134MiB/100%; cuda1=12677MiB/91%; time=2026-08-19T17:12:28+08:00260819-17:12:29 (UTC+8)
381remaining=454; core_remaining=231; overflow_remaining=223; running=2; done=452; failed=0; total=906; cuda0=14134MiB/100%; cuda1=12677MiB/100%; time=2026-08-19T17:02:00+08:00260819-17:02:03 (UTC+8)
380remaining=454; core_remaining=231; overflow_remaining=223; running=2; done=452; failed=0; total=906; cuda0=14134MiB/94%; cuda1=12677MiB/100%; time=2026-08-19T16:51:34+08:00260819-16:51:37 (UTC+8)
379remaining=455; core_remaining=232; overflow_remaining=223; running=2; done=451; failed=0; total=906; cuda0=14134MiB/100%; cuda1=12677MiB/100%; time=2026-08-19T16:41:09+08:00260819-16:41:11 (UTC+8)
378remaining=456; core_remaining=233; overflow_remaining=223; running=2; done=450; failed=0; total=906; cuda0=14136MiB/100%; cuda1=12677MiB/95%; time=2026-08-19T16:30:43+08:00260819-16:30:45 (UTC+8)
377remaining=456; core_remaining=233; overflow_remaining=223; running=2; done=450; failed=0; total=906; cuda0=14136MiB/86%; cuda1=12677MiB/100%; time=2026-08-19T16:20:16+08:00260819-16:20:18 (UTC+8)
376remaining=457; core_remaining=234; overflow_remaining=223; running=2; done=449; failed=0; total=906; cuda0=14136MiB/100%; cuda1=12677MiB/77%; time=2026-08-19T16:09:12+08:00260819-16:09:52 (UTC+8)
375remaining=458; core_remaining=235; overflow_remaining=223; running=2; done=448; failed=0; total=906; cuda0=14138MiB/91%; cuda1=12677MiB/96%; time=2026-08-19T15:59:11+08:00260819-15:59:26 (UTC+8)
374remaining=458; core_remaining=235; overflow_remaining=223; running=2; done=448; failed=0; total=906; cuda0=14138MiB/100%; cuda1=12677MiB/100%; time=2026-08-19T15:48:58+08:00260819-15:49:00 (UTC+8)
373remaining=459; core_remaining=236; overflow_remaining=223; running=2; done=447; failed=0; total=906; cuda0=14138MiB/100%; cuda1=12731MiB/100%; time=2026-08-19T15:38:33+08:00260819-15:38:34 (UTC+8)
372remaining=460; core_remaining=237; overflow_remaining=223; running=2; done=446; failed=0; total=906; cuda0=14188MiB/92%; cuda1=12731MiB/90%; time=2026-08-19T15:28:06+08:00260819-15:28:07 (UTC+8)
371remaining=460; core_remaining=237; overflow_remaining=223; running=2; done=446; failed=0; total=906; cuda0=14188MiB/69%; cuda1=12731MiB/86%; time=2026-08-19T15:17:40+08:00260819-15:17:41 (UTC+8)
370remaining=462; core_remaining=239; overflow_remaining=223; running=2; done=444; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/95%; time=2026-08-19T15:07:13+08:00260819-15:07:15 (UTC+8)
369remaining=462; core_remaining=239; overflow_remaining=223; running=2; done=444; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/92%; time=2026-08-19T14:56:13+08:00260819-14:56:48 (UTC+8)
368remaining=462; core_remaining=239; overflow_remaining=223; running=2; done=444; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T14:46:12+08:00260819-14:46:22 (UTC+8)
367remaining=462; core_remaining=239; overflow_remaining=223; running=2; done=444; failed=0; total=906; cuda0=14448MiB/92%; cuda1=12991MiB/100%; time=2026-08-19T14:35:54+08:00260819-14:35:56 (UTC+8)
366remaining=463; core_remaining=240; overflow_remaining=223; running=2; done=443; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T14:25:27+08:00260819-14:25:29 (UTC+8)
365remaining=464; core_remaining=241; overflow_remaining=223; running=2; done=442; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/93%; time=2026-08-19T14:15:02+08:00260819-14:15:03 (UTC+8)
364remaining=464; core_remaining=241; overflow_remaining=223; running=2; done=442; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T14:04:36+08:00260819-14:04:37 (UTC+8)
363remaining=464; core_remaining=241; overflow_remaining=223; running=2; done=442; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/13%; time=2026-08-19T13:54:08+08:00260819-13:54:10 (UTC+8)
362remaining=465; core_remaining=242; overflow_remaining=223; running=2; done=441; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/92%; time=2026-08-19T13:43:42+08:00260819-13:43:44 (UTC+8)
361remaining=466; core_remaining=243; overflow_remaining=223; running=2; done=440; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T13:33:17+08:00260819-13:33:18 (UTC+8)
360remaining=466; core_remaining=243; overflow_remaining=223; running=2; done=440; failed=0; total=906; cuda0=14448MiB/91%; cuda1=12991MiB/100%; time=2026-08-19T13:22:51+08:00260819-13:22:52 (UTC+8)
359remaining=466; core_remaining=243; overflow_remaining=223; running=2; done=440; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T13:12:25+08:00260819-13:12:26 (UTC+8)
358remaining=467; core_remaining=244; overflow_remaining=223; running=2; done=439; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12993MiB/100%; time=2026-08-19T13:01:59+08:00260819-13:02:00 (UTC+8)
357remaining=468; core_remaining=245; overflow_remaining=223; running=2; done=438; failed=0; total=906; cuda0=14448MiB/96%; cuda1=12993MiB/91%; time=2026-08-19T12:51:32+08:00260819-12:51:34 (UTC+8)
356remaining=468; core_remaining=245; overflow_remaining=223; running=2; done=438; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12993MiB/94%; time=2026-08-19T12:41:07+08:00260819-12:41:08 (UTC+8)
355remaining=468; core_remaining=245; overflow_remaining=223; running=2; done=438; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12993MiB/90%; time=2026-08-19T12:30:40+08:00260819-12:30:42 (UTC+8)
354remaining=469; core_remaining=246; overflow_remaining=223; running=2; done=437; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/92%; time=2026-08-19T12:20:14+08:00260819-12:20:16 (UTC+8)
353remaining=470; core_remaining=247; overflow_remaining=223; running=2; done=436; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T12:09:49+08:00260819-12:09:50 (UTC+8)
352remaining=470; core_remaining=247; overflow_remaining=223; running=2; done=436; failed=0; total=906; cuda0=14448MiB/97%; cuda1=12991MiB/100%; time=2026-08-19T11:59:22+08:00260819-11:59:24 (UTC+8)
351remaining=470; core_remaining=247; overflow_remaining=223; running=2; done=436; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/96%; time=2026-08-19T11:48:55+08:00260819-11:48:57 (UTC+8)
350remaining=470; core_remaining=247; overflow_remaining=223; running=2; done=436; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T11:38:30+08:00260819-11:38:31 (UTC+8)
349remaining=472; core_remaining=249; overflow_remaining=223; running=2; done=434; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T11:28:04+08:00260819-11:28:05 (UTC+8)
348remaining=472; core_remaining=249; overflow_remaining=223; running=2; done=434; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T11:17:38+08:00260819-11:17:40 (UTC+8)
347remaining=472; core_remaining=249; overflow_remaining=223; running=2; done=434; failed=0; total=906; cuda0=14448MiB/0%; cuda1=12991MiB/100%; time=2026-08-19T11:07:12+08:00260819-11:07:13 (UTC+8)
346remaining=472; core_remaining=249; overflow_remaining=223; running=2; done=434; failed=0; total=906; cuda0=14448MiB/96%; cuda1=12991MiB/96%; time=2026-08-19T10:56:46+08:00260819-10:56:47 (UTC+8)
345remaining=474; core_remaining=251; overflow_remaining=223; running=2; done=432; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12993MiB/100%; time=2026-08-19T10:46:19+08:00260819-10:46:21 (UTC+8)
344remaining=474; core_remaining=251; overflow_remaining=223; running=2; done=432; failed=0; total=906; cuda0=14448MiB/96%; cuda1=12993MiB/96%; time=2026-08-19T10:35:53+08:00260819-10:35:55 (UTC+8)
343remaining=474; core_remaining=251; overflow_remaining=223; running=2; done=432; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12993MiB/100%; time=2026-08-19T10:25:27+08:00260819-10:25:28 (UTC+8)
342remaining=474; core_remaining=251; overflow_remaining=223; running=2; done=432; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12993MiB/94%; time=2026-08-19T10:15:00+08:00260819-10:15:02 (UTC+8)
341remaining=476; core_remaining=253; overflow_remaining=223; running=2; done=430; failed=0; total=906; cuda0=14450MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T10:03:51+08:00260819-10:04:36 (UTC+8)
340remaining=476; core_remaining=253; overflow_remaining=223; running=2; done=430; failed=0; total=906; cuda0=14450MiB/100%; cuda1=12991MiB/91%; time=2026-08-19T09:53:50+08:00260819-09:54:10 (UTC+8)
339remaining=476; core_remaining=253; overflow_remaining=223; running=2; done=430; failed=0; total=906; cuda0=14450MiB/100%; cuda1=12991MiB/92%; time=2026-08-19T09:43:41+08:00260819-09:43:43 (UTC+8)
338remaining=476; core_remaining=253; overflow_remaining=223; running=2; done=430; failed=0; total=906; cuda0=14450MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T09:33:15+08:00260819-09:33:17 (UTC+8)
337remaining=477; core_remaining=254; overflow_remaining=223; running=2; done=429; failed=0; total=906; cuda0=14188MiB/24%; cuda1=12991MiB/95%; time=2026-08-19T09:22:49+08:00260819-09:22:51 (UTC+8)
336remaining=478; core_remaining=255; overflow_remaining=223; running=2; done=428; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/96%; time=2026-08-19T09:12:22+08:00260819-09:12:25 (UTC+8)
335remaining=478; core_remaining=255; overflow_remaining=223; running=2; done=428; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T09:01:58+08:00260819-09:01:59 (UTC+8)
334remaining=478; core_remaining=255; overflow_remaining=223; running=2; done=428; failed=0; total=906; cuda0=14448MiB/91%; cuda1=12991MiB/100%; time=2026-08-19T08:51:31+08:00260819-08:51:32 (UTC+8)
333remaining=479; core_remaining=256; overflow_remaining=223; running=2; done=427; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T08:41:05+08:00260819-08:41:06 (UTC+8)
332remaining=480; core_remaining=257; overflow_remaining=223; running=2; done=426; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/94%; time=2026-08-19T08:30:40+08:00260819-08:30:40 (UTC+8)
331remaining=480; core_remaining=257; overflow_remaining=223; running=2; done=426; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T08:20:13+08:00260819-08:20:14 (UTC+8)
330remaining=480; core_remaining=257; overflow_remaining=223; running=2; done=426; failed=0; total=906; cuda0=14448MiB/95%; cuda1=12991MiB/100%; time=2026-08-19T08:09:47+08:00260819-08:09:49 (UTC+8)
329remaining=481; core_remaining=258; overflow_remaining=223; running=2; done=425; failed=0; total=906; cuda0=14448MiB/93%; cuda1=12991MiB/100%; time=2026-08-19T07:59:21+08:00260819-07:59:22 (UTC+8)
328remaining=482; core_remaining=259; overflow_remaining=223; running=2; done=424; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/93%; time=2026-08-19T07:48:55+08:00260819-07:48:56 (UTC+8)
327remaining=482; core_remaining=259; overflow_remaining=223; running=2; done=424; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/93%; time=2026-08-19T07:38:29+08:00260819-07:38:30 (UTC+8)
326remaining=482; core_remaining=259; overflow_remaining=223; running=2; done=424; failed=0; total=906; cuda0=14448MiB/92%; cuda1=12991MiB/97%; time=2026-08-19T07:28:02+08:00260819-07:28:04 (UTC+8)
325remaining=483; core_remaining=260; overflow_remaining=223; running=2; done=423; failed=0; total=906; cuda0=14448MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T07:17:35+08:00260819-07:17:38 (UTC+8)
324remaining=484; core_remaining=261; overflow_remaining=223; running=2; done=422; failed=0; total=906; cuda0=14450MiB/90%; cuda1=12991MiB/100%; time=2026-08-19T07:07:10+08:00260819-07:07:11 (UTC+8)
323remaining=484; core_remaining=261; overflow_remaining=223; running=2; done=422; failed=0; total=906; cuda0=14450MiB/93%; cuda1=12991MiB/100%; time=2026-08-19T06:56:43+08:00260819-06:56:45 (UTC+8)
322remaining=484; core_remaining=261; overflow_remaining=223; running=2; done=422; failed=0; total=906; cuda0=14450MiB/100%; cuda1=12991MiB/100%; time=2026-08-19T06:46:18+08:00260819-06:46:19 (UTC+8)
321remaining=484; core_remaining=261; overflow_remaining=223; running=2; done=422; failed=0; total=906; cuda0=14450MiB/95%; cuda1=12991MiB/100%; time=2026-08-19T06:35:51+08:00260819-06:35:53 (UTC+8)
320remaining=486; core_remaining=263; overflow_remaining=223; running=2; done=420; failed=0; total=906; cuda0=14452MiB/91%; cuda1=12993MiB/100%; time=2026-08-19T06:25:26+08:00260819-06:25:27 (UTC+8)
319remaining=486; core_remaining=263; overflow_remaining=223; running=2; done=420; failed=0; total=906; cuda0=14452MiB/100%; cuda1=12993MiB/100%; time=2026-08-19T06:15:00+08:00260819-06:15:01 (UTC+8)
318remaining=486; core_remaining=263; overflow_remaining=223; running=2; done=420; failed=0; total=906; cuda0=14452MiB/100%; cuda1=12993MiB/100%; time=2026-08-19T06:04:34+08:00260819-06:04:35 (UTC+8)
317remaining=486; core_remaining=263; overflow_remaining=223; running=2; done=420; failed=0; total=906; cuda0=14452MiB/100%; cuda1=12993MiB/95%; time=2026-08-19T05:54:08+08:00260819-05:54:09 (UTC+8)
316remaining=488; core_remaining=265; overflow_remaining=223; running=2; done=418; failed=0; total=906; cuda0=14456MiB/100%; cuda1=12995MiB/91%; time=2026-08-19T05:43:42+08:00260819-05:43:44 (UTC+8)
315remaining=488; core_remaining=265; overflow_remaining=223; running=2; done=418; failed=0; total=906; cuda0=14456MiB/93%; cuda1=12995MiB/91%; time=2026-08-19T05:32:42+08:00260819-05:33:18 (UTC+8)
314remaining=488; core_remaining=265; overflow_remaining=223; running=2; done=418; failed=0; total=906; cuda0=14456MiB/92%; cuda1=12995MiB/100%; time=2026-08-19T05:22:41+08:00260819-05:22:52 (UTC+8)
313remaining=488; core_remaining=265; overflow_remaining=223; running=2; done=418; failed=0; total=906; cuda0=14456MiB/91%; cuda1=12995MiB/100%; time=2026-08-19T05:12:25+08:00260819-05:12:26 (UTC+8)
312remaining=490; core_remaining=267; overflow_remaining=223; running=2; done=416; failed=0; total=906; cuda0=14408MiB/100%; cuda1=12997MiB/89%; time=2026-08-19T05:01:59+08:00260819-05:02:00 (UTC+8)
311remaining=490; core_remaining=267; overflow_remaining=223; running=2; done=416; failed=0; total=906; cuda0=14408MiB/100%; cuda1=12997MiB/100%; time=2026-08-19T04:50:48+08:00260819-04:51:34 (UTC+8)
310remaining=490; core_remaining=267; overflow_remaining=223; running=2; done=416; failed=0; total=906; cuda0=14408MiB/100%; cuda1=12997MiB/93%; time=2026-08-19T04:40:47+08:00260819-04:41:08 (UTC+8)
309remaining=490; core_remaining=267; overflow_remaining=223; running=2; done=416; failed=0; total=906; cuda0=14408MiB/100%; cuda1=12997MiB/96%; time=2026-08-19T04:30:40+08:00260819-04:30:41 (UTC+8)
308remaining=492; core_remaining=269; overflow_remaining=223; running=2; done=414; failed=0; total=906; cuda0=13846MiB/100%; cuda1=12949MiB/100%; time=2026-08-19T04:20:14+08:00260819-04:20:15 (UTC+8)
307remaining=492; core_remaining=269; overflow_remaining=223; running=2; done=414; failed=0; total=906; cuda0=13846MiB/66%; cuda1=12949MiB/96%; time=2026-08-19T04:09:48+08:00260819-04:09:49 (UTC+8)
306remaining=492; core_remaining=269; overflow_remaining=223; running=2; done=414; failed=0; total=906; cuda0=13846MiB/91%; cuda1=12949MiB/100%; time=2026-08-19T03:59:21+08:00260819-03:59:23 (UTC+8)
305remaining=492; core_remaining=269; overflow_remaining=223; running=2; done=414; failed=0; total=906; cuda0=13846MiB/100%; cuda1=12949MiB/96%; time=2026-08-19T03:48:55+08:00260819-03:48:57 (UTC+8)
304remaining=494; core_remaining=271; overflow_remaining=223; running=2; done=412; failed=0; total=906; cuda0=13850MiB/100%; cuda1=12391MiB/100%; time=2026-08-19T03:38:29+08:00260819-03:38:31 (UTC+8)
303remaining=494; core_remaining=271; overflow_remaining=223; running=2; done=412; failed=0; total=906; cuda0=13850MiB/92%; cuda1=12391MiB/100%; time=2026-08-19T03:28:04+08:00260819-03:28:05 (UTC+8)
302remaining=494; core_remaining=271; overflow_remaining=223; running=2; done=412; failed=0; total=906; cuda0=13850MiB/94%; cuda1=12391MiB/100%; time=2026-08-19T03:17:37+08:00260819-03:17:38 (UTC+8)
301remaining=494; core_remaining=271; overflow_remaining=223; running=2; done=412; failed=0; total=906; cuda0=13850MiB/100%; cuda1=12391MiB/100%; time=2026-08-19T03:07:11+08:00260819-03:07:12 (UTC+8)
300remaining=496; core_remaining=273; overflow_remaining=223; running=2; done=410; failed=0; total=906; cuda0=13848MiB/100%; cuda1=12389MiB/100%; time=2026-08-19T02:56:44+08:00260819-02:56:46 (UTC+8)
299remaining=496; core_remaining=273; overflow_remaining=223; running=2; done=410; failed=0; total=906; cuda0=13848MiB/100%; cuda1=12389MiB/100%; time=2026-08-19T02:35:01+08:00260819-02:35:44 (UTC+8)
298remaining=496; core_remaining=273; overflow_remaining=223; running=2; done=410; failed=0; total=906; cuda0=13848MiB/100%; cuda1=12389MiB/100%; time=2026-08-19T02:25:00+08:00260819-02:25:18 (UTC+8)
297remaining=498; core_remaining=275; overflow_remaining=223; running=2; done=408; failed=0; total=906; cuda0=13846MiB/92%; cuda1=12393MiB/95%; time=2026-08-19T02:14:50+08:00260819-02:14:52 (UTC+8)
296remaining=498; core_remaining=275; overflow_remaining=223; running=2; done=408; failed=0; total=906; cuda0=13846MiB/100%; cuda1=12393MiB/99%; time=2026-08-19T02:04:24+08:00260819-02:04:26 (UTC+8)
295remaining=498; core_remaining=275; overflow_remaining=223; running=2; done=408; failed=0; total=906; cuda0=13846MiB/94%; cuda1=12393MiB/91%; time=2026-08-19T01:53:06+08:00260819-01:54:00 (UTC+8)
294remaining=499; core_remaining=276; overflow_remaining=223; running=2; done=407; failed=0; total=906; cuda0=13846MiB/100%; cuda1=12391MiB/100%; time=2026-08-19T01:43:05+08:00260819-01:43:34 (UTC+8)
293remaining=500; core_remaining=277; overflow_remaining=223; running=2; done=406; failed=0; total=906; cuda0=13980MiB/92%; cuda1=12391MiB/100%; time=2026-08-19T01:33:04+08:00260819-01:33:08 (UTC+8)
292remaining=500; core_remaining=277; overflow_remaining=223; running=2; done=406; failed=0; total=906; cuda0=13980MiB/100%; cuda1=12391MiB/100%; time=2026-08-19T01:22:41+08:00260819-01:22:41 (UTC+8)
291remaining=500; core_remaining=277; overflow_remaining=223; running=2; done=406; failed=0; total=906; cuda0=13980MiB/94%; cuda1=12391MiB/100%; time=2026-08-19T01:12:13+08:00260819-01:12:15 (UTC+8)
290remaining=502; core_remaining=279; overflow_remaining=223; running=2; done=404; failed=0; total=906; cuda0=14194MiB/100%; cuda1=12389MiB/100%; time=2026-08-19T01:01:08+08:00260819-01:01:49 (UTC+8)
289remaining=502; core_remaining=279; overflow_remaining=223; running=2; done=404; failed=0; total=906; cuda0=14194MiB/100%; cuda1=12389MiB/92%; time=2026-08-19T00:51:07+08:00260819-00:51:23 (UTC+8)
288remaining=502; core_remaining=279; overflow_remaining=223; running=2; done=404; failed=0; total=906; cuda0=14194MiB/100%; cuda1=12389MiB/100%; time=2026-08-19T00:39:58+08:00260819-00:40:57 (UTC+8)
287remaining=502; core_remaining=279; overflow_remaining=223; running=2; done=404; failed=0; total=906; cuda0=14194MiB/100%; cuda1=12389MiB/92%; time=2026-08-19T00:29:57+08:00260819-00:30:31 (UTC+8)
286remaining=504; core_remaining=281; overflow_remaining=223; running=2; done=402; failed=0; total=906; cuda0=14192MiB/100%; cuda1=12393MiB/100%; time=2026-08-19T00:19:56+08:00260819-00:20:05 (UTC+8)
285remaining=504; core_remaining=281; overflow_remaining=223; running=2; done=402; failed=0; total=906; cuda0=14192MiB/97%; cuda1=12393MiB/100%; time=2026-08-19T00:09:37+08:00260819-00:09:38 (UTC+8)
284remaining=504; core_remaining=281; overflow_remaining=223; running=2; done=402; failed=0; total=906; cuda0=14192MiB/100%; cuda1=12393MiB/100%; time=2026-08-18T23:59:10+08:00260818-23:59:12 (UTC+8)
283remaining=504; core_remaining=281; overflow_remaining=223; running=2; done=402; failed=0; total=906; cuda0=14192MiB/100%; cuda1=12393MiB/100%; time=2026-08-18T23:48:42+08:00260818-23:48:45 (UTC+8)
282remaining=506; core_remaining=283; overflow_remaining=223; running=2; done=400; failed=0; total=906; cuda0=14300MiB/100%; cuda1=12391MiB/100%; time=2026-08-18T23:38:14+08:00260818-23:38:19 (UTC+8)
281remaining=506; core_remaining=283; overflow_remaining=223; running=2; done=400; failed=0; total=906; cuda0=14300MiB/100%; cuda1=12391MiB/92%; time=2026-08-18T23:27:48+08:00260818-23:27:50 (UTC+8)
280remaining=506; core_remaining=283; overflow_remaining=223; running=2; done=400; failed=0; total=906; cuda0=14300MiB/100%; cuda1=12391MiB/93%; time=2026-08-18T23:17:22+08:00260818-23:17:24 (UTC+8)
279remaining=506; core_remaining=283; overflow_remaining=223; running=2; done=400; failed=0; total=906; cuda0=14300MiB/92%; cuda1=12391MiB/91%; time=2026-08-18T23:06:56+08:00260818-23:06:57 (UTC+8)
278remaining=508; core_remaining=285; overflow_remaining=223; running=2; done=398; failed=0; total=906; cuda0=14298MiB/96%; cuda1=12389MiB/100%; time=2026-08-18T22:56:29+08:00260818-22:56:30 (UTC+8)
277remaining=508; core_remaining=285; overflow_remaining=223; running=2; done=398; failed=0; total=906; cuda0=14298MiB/95%; cuda1=12389MiB/42%; time=2026-08-18T22:46:02+08:00260818-22:46:04 (UTC+8)
276remaining=508; core_remaining=285; overflow_remaining=223; running=2; done=398; failed=0; total=906; cuda0=14298MiB/100%; cuda1=12389MiB/100%; time=2026-08-18T22:35:37+08:00260818-22:35:38 (UTC+8)
275remaining=508; core_remaining=285; overflow_remaining=223; running=2; done=398; failed=0; total=906; cuda0=14298MiB/100%; cuda1=12389MiB/100%; time=2026-08-18T22:25:10+08:00260818-22:25:11 (UTC+8)
274remaining=510; core_remaining=287; overflow_remaining=223; running=2; done=396; failed=0; total=906; cuda0=9758MiB/100%; cuda1=7975MiB/96%; time=2026-08-18T22:14:44+08:00260818-22:14:45 (UTC+8)
273remaining=510; core_remaining=287; overflow_remaining=223; running=2; done=396; failed=0; total=906; cuda0=9758MiB/91%; cuda1=7975MiB/100%; time=2026-08-18T22:04:18+08:00260818-22:04:19 (UTC+8)
272remaining=510; core_remaining=287; overflow_remaining=223; running=2; done=396; failed=0; total=906; cuda0=9758MiB/100%; cuda1=7975MiB/100%; time=2026-08-18T21:53:52+08:00260818-21:53:53 (UTC+8)
271remaining=511; core_remaining=288; overflow_remaining=223; running=1; done=395; failed=0; total=906; cuda0=1737MiB/0%; cuda1=11445MiB/100%; time=2026-08-18T21:43:25+08:00260818-21:43:27 (UTC+8)
270remaining=247; core_remaining=120; overflow_remaining=127; running=2; done=394; failed=0; total=641; cuda0=13168MiB/100%; cuda1=11445MiB/100%; time=2026-08-18T21:19:54+08:00260818-21:19:56 (UTC+8)
269remaining=247; core_remaining=120; overflow_remaining=127; running=2; done=394; failed=0; total=641; cuda0=13168MiB/100%; cuda1=11445MiB/100%; time=2026-08-18T21:09:28+08:00260818-21:09:29 (UTC+8)
268remaining=247; core_remaining=120; overflow_remaining=127; running=2; done=394; failed=0; total=641; cuda0=13166MiB/100%; cuda1=11445MiB/100%; time=2026-08-18T18:51:28+08:00260818-20:59:03 (UTC+8)
267remaining=247; core_remaining=120; overflow_remaining=127; running=2; done=394; failed=0; total=641; cuda0=13166MiB/95%; cuda1=11445MiB/100%; time=2026-08-18T18:41:27+08:00260818-20:48:37 (UTC+8)
266remaining=466; core_remaining=339; overflow_remaining=127; running=2; done=175; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T16:08:41+08:00260816-16:09:35 (UTC+8)
265remaining=467; core_remaining=340; overflow_remaining=127; running=2; done=174; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/100%; time=2026-08-16T15:58:40+08:00260816-15:59:09 (UTC+8)
264remaining=467; core_remaining=340; overflow_remaining=127; running=2; done=174; failed=0; total=641; cuda0=9328MiB/94%; cuda1=7607MiB/100%; time=2026-08-16T15:48:39+08:00260816-15:48:43 (UTC+8)
263remaining=469; core_remaining=342; overflow_remaining=127; running=2; done=172; failed=0; total=641; cuda0=9328MiB/60%; cuda1=7607MiB/100%; time=2026-08-16T15:38:15+08:00260816-15:38:17 (UTC+8)
262remaining=469; core_remaining=342; overflow_remaining=127; running=2; done=172; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/97%; time=2026-08-16T15:27:50+08:00260816-15:27:51 (UTC+8)
261remaining=469; core_remaining=342; overflow_remaining=127; running=2; done=172; failed=0; total=641; cuda0=9328MiB/72%; cuda1=7607MiB/72%; time=2026-08-16T15:17:25+08:00260816-15:17:25 (UTC+8)
260remaining=471; core_remaining=344; overflow_remaining=127; running=2; done=170; failed=0; total=641; cuda0=9328MiB/89%; cuda1=7607MiB/82%; time=2026-08-16T15:06:58+08:00260816-15:07:00 (UTC+8)
259remaining=471; core_remaining=344; overflow_remaining=127; running=2; done=170; failed=0; total=641; cuda0=9328MiB/78%; cuda1=7607MiB/95%; time=2026-08-16T14:56:33+08:00260816-14:56:34 (UTC+8)
258remaining=472; core_remaining=345; overflow_remaining=127; running=2; done=169; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/88%; time=2026-08-16T14:46:06+08:00260816-14:46:08 (UTC+8)
257remaining=473; core_remaining=346; overflow_remaining=127; running=2; done=168; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/90%; time=2026-08-16T14:35:41+08:00260816-14:35:42 (UTC+8)
256remaining=473; core_remaining=346; overflow_remaining=127; running=2; done=168; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T14:25:15+08:00260816-14:25:16 (UTC+8)
255remaining=475; core_remaining=348; overflow_remaining=127; running=2; done=166; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T14:14:49+08:00260816-14:14:50 (UTC+8)
254remaining=475; core_remaining=348; overflow_remaining=127; running=2; done=166; failed=0; total=641; cuda0=9328MiB/26%; cuda1=7607MiB/100%; time=2026-08-16T14:04:23+08:00260816-14:04:25 (UTC+8)
253remaining=476; core_remaining=349; overflow_remaining=127; running=2; done=165; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T13:53:58+08:00260816-13:53:59 (UTC+8)
252remaining=477; core_remaining=350; overflow_remaining=127; running=2; done=164; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/90%; time=2026-08-16T13:43:32+08:00260816-13:43:33 (UTC+8)
251remaining=477; core_remaining=350; overflow_remaining=127; running=2; done=164; failed=0; total=641; cuda0=9328MiB/92%; cuda1=7607MiB/91%; time=2026-08-16T13:33:05+08:00260816-13:33:07 (UTC+8)
250remaining=479; core_remaining=352; overflow_remaining=127; running=2; done=162; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/91%; time=2026-08-16T13:22:40+08:00260816-13:22:41 (UTC+8)
249remaining=479; core_remaining=352; overflow_remaining=127; running=2; done=162; failed=0; total=641; cuda0=9328MiB/95%; cuda1=7607MiB/91%; time=2026-08-16T13:12:15+08:00260816-13:12:16 (UTC+8)
248remaining=480; core_remaining=353; overflow_remaining=127; running=2; done=161; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/94%; time=2026-08-16T13:01:48+08:00260816-13:01:50 (UTC+8)
247remaining=481; core_remaining=354; overflow_remaining=127; running=2; done=160; failed=0; total=641; cuda0=9328MiB/96%; cuda1=7607MiB/90%; time=2026-08-16T12:51:23+08:00260816-12:51:24 (UTC+8)
246remaining=481; core_remaining=354; overflow_remaining=127; running=2; done=160; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T12:40:56+08:00260816-12:40:58 (UTC+8)
245remaining=483; core_remaining=356; overflow_remaining=127; running=2; done=158; failed=0; total=641; cuda0=9328MiB/93%; cuda1=7607MiB/100%; time=2026-08-16T12:30:31+08:00260816-12:30:32 (UTC+8)
244remaining=483; core_remaining=356; overflow_remaining=127; running=2; done=158; failed=0; total=641; cuda0=9328MiB/79%; cuda1=7607MiB/90%; time=2026-08-16T12:19:19+08:00260816-12:20:06 (UTC+8)
243remaining=484; core_remaining=357; overflow_remaining=127; running=2; done=157; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/100%; time=2026-08-16T12:09:18+08:00260816-12:09:41 (UTC+8)
242remaining=485; core_remaining=358; overflow_remaining=127; running=2; done=156; failed=0; total=641; cuda0=9328MiB/79%; cuda1=7607MiB/93%; time=2026-08-16T11:59:13+08:00260816-11:59:15 (UTC+8)
241remaining=485; core_remaining=358; overflow_remaining=127; running=2; done=156; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T11:48:48+08:00260816-11:48:49 (UTC+8)
240remaining=486; core_remaining=359; overflow_remaining=127; running=2; done=155; failed=0; total=641; cuda0=9328MiB/89%; cuda1=7607MiB/100%; time=2026-08-16T11:38:21+08:00260816-11:38:23 (UTC+8)
239remaining=487; core_remaining=360; overflow_remaining=127; running=2; done=154; failed=0; total=641; cuda0=9328MiB/94%; cuda1=7607MiB/100%; time=2026-08-16T11:27:56+08:00260816-11:27:57 (UTC+8)
238remaining=487; core_remaining=360; overflow_remaining=127; running=2; done=154; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/96%; time=2026-08-16T11:17:31+08:00260816-11:17:31 (UTC+8)
237remaining=489; core_remaining=362; overflow_remaining=127; running=2; done=152; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T11:07:04+08:00260816-11:07:05 (UTC+8)
236remaining=489; core_remaining=362; overflow_remaining=127; running=2; done=152; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/100%; time=2026-08-16T10:56:38+08:00260816-10:56:40 (UTC+8)
235remaining=490; core_remaining=363; overflow_remaining=127; running=2; done=151; failed=0; total=641; cuda0=9328MiB/85%; cuda1=7607MiB/100%; time=2026-08-16T10:46:12+08:00260816-10:46:14 (UTC+8)
234remaining=491; core_remaining=364; overflow_remaining=127; running=2; done=150; failed=0; total=641; cuda0=9328MiB/93%; cuda1=7607MiB/95%; time=2026-08-16T10:35:47+08:00260816-10:35:48 (UTC+8)
233remaining=491; core_remaining=364; overflow_remaining=127; running=2; done=150; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T10:25:20+08:00260816-10:25:22 (UTC+8)
232remaining=493; core_remaining=366; overflow_remaining=127; running=2; done=148; failed=0; total=641; cuda0=9328MiB/97%; cuda1=7607MiB/92%; time=2026-08-16T10:14:55+08:00260816-10:14:56 (UTC+8)
231remaining=493; core_remaining=366; overflow_remaining=127; running=2; done=148; failed=0; total=641; cuda0=9328MiB/95%; cuda1=7607MiB/73%; time=2026-08-16T10:04:30+08:00260816-10:04:30 (UTC+8)
230remaining=494; core_remaining=367; overflow_remaining=127; running=2; done=147; failed=0; total=641; cuda0=9328MiB/93%; cuda1=7607MiB/79%; time=2026-08-16T09:54:03+08:00260816-09:54:05 (UTC+8)
229remaining=495; core_remaining=368; overflow_remaining=127; running=2; done=146; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/84%; time=2026-08-16T09:43:38+08:00260816-09:43:39 (UTC+8)
228remaining=495; core_remaining=368; overflow_remaining=127; running=2; done=146; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T09:33:11+08:00260816-09:33:13 (UTC+8)
227remaining=497; core_remaining=370; overflow_remaining=127; running=2; done=144; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T09:22:46+08:00260816-09:22:47 (UTC+8)
226remaining=497; core_remaining=370; overflow_remaining=127; running=2; done=144; failed=0; total=641; cuda0=9328MiB/90%; cuda1=7607MiB/100%; time=2026-08-16T09:11:25+08:00260816-09:12:21 (UTC+8)
225remaining=498; core_remaining=371; overflow_remaining=127; running=2; done=143; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/81%; time=2026-08-16T09:01:24+08:00260816-09:01:55 (UTC+8)
224remaining=499; core_remaining=372; overflow_remaining=127; running=2; done=142; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T08:51:24+08:00260816-08:51:30 (UTC+8)
223remaining=499; core_remaining=372; overflow_remaining=127; running=2; done=142; failed=0; total=641; cuda0=9328MiB/84%; cuda1=7607MiB/100%; time=2026-08-16T08:41:02+08:00260816-08:41:04 (UTC+8)
222remaining=500; core_remaining=373; overflow_remaining=127; running=2; done=141; failed=0; total=641; cuda0=9328MiB/87%; cuda1=7607MiB/90%; time=2026-08-16T08:30:37+08:00260816-08:30:38 (UTC+8)
221remaining=501; core_remaining=374; overflow_remaining=127; running=2; done=140; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/100%; time=2026-08-16T08:20:10+08:00260816-08:20:12 (UTC+8)
220remaining=501; core_remaining=374; overflow_remaining=127; running=2; done=140; failed=0; total=641; cuda0=9328MiB/90%; cuda1=7607MiB/95%; time=2026-08-16T08:09:45+08:00260816-08:09:46 (UTC+8)
219remaining=502; core_remaining=375; overflow_remaining=127; running=2; done=139; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/100%; time=2026-08-16T07:59:20+08:00260816-07:59:21 (UTC+8)
218remaining=503; core_remaining=376; overflow_remaining=127; running=2; done=138; failed=0; total=641; cuda0=9328MiB/90%; cuda1=7607MiB/96%; time=2026-08-16T07:48:53+08:00260816-07:48:55 (UTC+8)
217remaining=503; core_remaining=376; overflow_remaining=127; running=2; done=138; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T07:38:28+08:00260816-07:38:29 (UTC+8)
216remaining=504; core_remaining=377; overflow_remaining=127; running=2; done=137; failed=0; total=641; cuda0=9328MiB/83%; cuda1=7607MiB/9%; time=2026-08-16T07:28:01+08:00260816-07:28:03 (UTC+8)
215remaining=505; core_remaining=378; overflow_remaining=127; running=2; done=136; failed=0; total=641; cuda0=9328MiB/90%; cuda1=7607MiB/100%; time=2026-08-16T07:17:36+08:00260816-07:17:37 (UTC+8)
214remaining=505; core_remaining=378; overflow_remaining=127; running=2; done=136; failed=0; total=641; cuda0=9328MiB/94%; cuda1=7607MiB/91%; time=2026-08-16T07:07:11+08:00260816-07:07:11 (UTC+8)
213remaining=507; core_remaining=380; overflow_remaining=127; running=2; done=134; failed=0; total=641; cuda0=9328MiB/96%; cuda1=7607MiB/93%; time=2026-08-16T06:56:45+08:00260816-06:56:46 (UTC+8)
212remaining=507; core_remaining=380; overflow_remaining=127; running=2; done=134; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/89%; time=2026-08-16T06:46:18+08:00260816-06:46:20 (UTC+8)
211remaining=507; core_remaining=380; overflow_remaining=127; running=2; done=134; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/92%; time=2026-08-16T06:35:53+08:00260816-06:35:54 (UTC+8)
210remaining=509; core_remaining=382; overflow_remaining=127; running=2; done=132; failed=0; total=641; cuda0=9328MiB/97%; cuda1=7607MiB/100%; time=2026-08-16T06:25:27+08:00260816-06:25:28 (UTC+8)
209remaining=509; core_remaining=382; overflow_remaining=127; running=2; done=132; failed=0; total=641; cuda0=9328MiB/98%; cuda1=7607MiB/92%; time=2026-08-16T06:15:01+08:00260816-06:15:02 (UTC+8)
208remaining=510; core_remaining=383; overflow_remaining=127; running=2; done=131; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/39%; time=2026-08-16T06:04:35+08:00260816-06:04:37 (UTC+8)
207remaining=511; core_remaining=384; overflow_remaining=127; running=2; done=130; failed=0; total=641; cuda0=9328MiB/89%; cuda1=7607MiB/94%; time=2026-08-16T05:53:30+08:00260816-05:54:11 (UTC+8)
206remaining=511; core_remaining=384; overflow_remaining=127; running=2; done=130; failed=0; total=641; cuda0=9328MiB/96%; cuda1=7607MiB/100%; time=2026-08-16T05:43:29+08:00260816-05:43:45 (UTC+8)
205remaining=512; core_remaining=385; overflow_remaining=127; running=2; done=129; failed=0; total=641; cuda0=9328MiB/92%; cuda1=7607MiB/79%; time=2026-08-16T05:33:17+08:00260816-05:33:19 (UTC+8)
204remaining=513; core_remaining=386; overflow_remaining=127; running=2; done=128; failed=0; total=641; cuda0=9328MiB/61%; cuda1=7607MiB/100%; time=2026-08-16T05:22:52+08:00260816-05:22:53 (UTC+8)
203remaining=513; core_remaining=386; overflow_remaining=127; running=2; done=128; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T05:12:26+08:00260816-05:12:27 (UTC+8)
202remaining=514; core_remaining=387; overflow_remaining=127; running=2; done=127; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/91%; time=2026-08-16T05:02:01+08:00260816-05:02:02 (UTC+8)
201remaining=515; core_remaining=388; overflow_remaining=127; running=2; done=126; failed=0; total=641; cuda0=9328MiB/64%; cuda1=7607MiB/100%; time=2026-08-16T04:51:35+08:00260816-04:51:36 (UTC+8)
200remaining=515; core_remaining=388; overflow_remaining=127; running=2; done=126; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T04:41:08+08:00260816-04:41:10 (UTC+8)
199remaining=516; core_remaining=389; overflow_remaining=127; running=2; done=125; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/95%; time=2026-08-16T04:30:42+08:00260816-04:30:44 (UTC+8)
198remaining=517; core_remaining=390; overflow_remaining=127; running=2; done=124; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/100%; time=2026-08-16T04:20:17+08:00260816-04:20:18 (UTC+8)
197remaining=517; core_remaining=390; overflow_remaining=127; running=2; done=124; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T04:09:50+08:00260816-04:09:52 (UTC+8)
196remaining=518; core_remaining=391; overflow_remaining=127; running=2; done=123; failed=0; total=641; cuda0=9328MiB/92%; cuda1=7607MiB/99%; time=2026-08-16T03:59:24+08:00260816-03:59:26 (UTC+8)
195remaining=519; core_remaining=392; overflow_remaining=127; running=2; done=122; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/95%; time=2026-08-16T03:48:59+08:00260816-03:49:01 (UTC+8)
194remaining=519; core_remaining=392; overflow_remaining=127; running=2; done=122; failed=0; total=641; cuda0=9328MiB/90%; cuda1=7607MiB/92%; time=2026-08-16T03:38:34+08:00260816-03:38:35 (UTC+8)
193remaining=520; core_remaining=393; overflow_remaining=127; running=2; done=121; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/96%; time=2026-08-16T03:28:07+08:00260816-03:28:09 (UTC+8)
192remaining=521; core_remaining=394; overflow_remaining=127; running=2; done=120; failed=0; total=641; cuda0=9328MiB/97%; cuda1=7607MiB/100%; time=2026-08-16T03:17:42+08:00260816-03:17:43 (UTC+8)
191remaining=521; core_remaining=394; overflow_remaining=127; running=2; done=120; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/100%; time=2026-08-16T03:07:16+08:00260816-03:07:17 (UTC+8)
190remaining=523; core_remaining=396; overflow_remaining=127; running=2; done=118; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/92%; time=2026-08-16T02:56:06+08:00260816-02:56:51 (UTC+8)
189remaining=523; core_remaining=396; overflow_remaining=127; running=2; done=118; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/96%; time=2026-08-16T02:46:05+08:00260816-02:46:25 (UTC+8)
188remaining=523; core_remaining=396; overflow_remaining=127; running=2; done=118; failed=0; total=641; cuda0=9328MiB/96%; cuda1=7607MiB/100%; time=2026-08-16T02:35:59+08:00260816-02:36:00 (UTC+8)
187remaining=525; core_remaining=398; overflow_remaining=127; running=2; done=116; failed=0; total=641; cuda0=9328MiB/90%; cuda1=7607MiB/100%; time=2026-08-16T02:25:33+08:00260816-02:25:34 (UTC+8)
186remaining=525; core_remaining=398; overflow_remaining=127; running=2; done=116; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T02:15:06+08:00260816-02:15:08 (UTC+8)
185remaining=526; core_remaining=399; overflow_remaining=127; running=2; done=115; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/96%; time=2026-08-16T02:04:41+08:00260816-02:04:42 (UTC+8)
184remaining=527; core_remaining=400; overflow_remaining=127; running=2; done=114; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T01:54:15+08:00260816-01:54:16 (UTC+8)
183remaining=527; core_remaining=400; overflow_remaining=127; running=2; done=114; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T01:43:49+08:00260816-01:43:50 (UTC+8)
182remaining=528; core_remaining=401; overflow_remaining=127; running=2; done=113; failed=0; total=641; cuda0=9328MiB/98%; cuda1=7607MiB/100%; time=2026-08-16T01:33:23+08:00260816-01:33:25 (UTC+8)
181remaining=529; core_remaining=402; overflow_remaining=127; running=2; done=112; failed=0; total=641; cuda0=9328MiB/88%; cuda1=7607MiB/100%; time=2026-08-16T01:22:58+08:00260816-01:22:59 (UTC+8)
180remaining=529; core_remaining=402; overflow_remaining=127; running=2; done=112; failed=0; total=641; cuda0=9328MiB/96%; cuda1=7607MiB/100%; time=2026-08-16T01:12:31+08:00260816-01:12:33 (UTC+8)
179remaining=530; core_remaining=403; overflow_remaining=127; running=2; done=111; failed=0; total=641; cuda0=9328MiB/91%; cuda1=7607MiB/97%; time=2026-08-16T01:02:06+08:00260816-01:02:07 (UTC+8)
178remaining=531; core_remaining=404; overflow_remaining=127; running=2; done=110; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T00:51:40+08:00260816-00:51:41 (UTC+8)
177remaining=532; core_remaining=405; overflow_remaining=127; running=2; done=109; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/96%; time=2026-08-16T00:40:30+08:00260816-00:41:16 (UTC+8)
176remaining=533; core_remaining=406; overflow_remaining=127; running=2; done=108; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/93%; time=2026-08-16T00:30:29+08:00260816-00:30:50 (UTC+8)
175remaining=533; core_remaining=406; overflow_remaining=127; running=2; done=108; failed=0; total=641; cuda0=9328MiB/96%; cuda1=7607MiB/93%; time=2026-08-16T00:20:23+08:00260816-00:20:24 (UTC+8)
174remaining=534; core_remaining=407; overflow_remaining=127; running=2; done=107; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-16T00:09:57+08:00260816-00:09:58 (UTC+8)
173remaining=535; core_remaining=408; overflow_remaining=127; running=2; done=106; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T23:59:31+08:00260815-23:59:32 (UTC+8)
172remaining=536; core_remaining=409; overflow_remaining=127; running=2; done=105; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T23:49:05+08:00260815-23:49:06 (UTC+8)
171remaining=537; core_remaining=410; overflow_remaining=127; running=2; done=104; failed=0; total=641; cuda0=9328MiB/82%; cuda1=7607MiB/100%; time=2026-08-15T23:38:39+08:00260815-23:38:41 (UTC+8)
170remaining=537; core_remaining=410; overflow_remaining=127; running=2; done=104; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/95%; time=2026-08-15T23:28:13+08:00260815-23:28:15 (UTC+8)
169remaining=538; core_remaining=411; overflow_remaining=127; running=2; done=103; failed=0; total=641; cuda0=9328MiB/89%; cuda1=7607MiB/100%; time=2026-08-15T23:17:48+08:00260815-23:17:49 (UTC+8)
168remaining=539; core_remaining=412; overflow_remaining=127; running=2; done=102; failed=0; total=641; cuda0=9328MiB/90%; cuda1=7607MiB/64%; time=2026-08-15T23:07:22+08:00260815-23:07:23 (UTC+8)
167remaining=539; core_remaining=412; overflow_remaining=127; running=2; done=102; failed=0; total=641; cuda0=9328MiB/95%; cuda1=7607MiB/100%; time=2026-08-15T22:56:56+08:00260815-22:56:57 (UTC+8)
166remaining=541; core_remaining=414; overflow_remaining=127; running=2; done=100; failed=0; total=641; cuda0=9328MiB/94%; cuda1=7607MiB/100%; time=2026-08-15T22:46:31+08:00260815-22:46:31 (UTC+8)
165remaining=541; core_remaining=414; overflow_remaining=127; running=2; done=100; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/82%; time=2026-08-15T22:36:04+08:00260815-22:36:06 (UTC+8)
164remaining=542; core_remaining=415; overflow_remaining=127; running=2; done=99; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/94%; time=2026-08-15T22:25:39+08:00260815-22:25:40 (UTC+8)
163remaining=543; core_remaining=416; overflow_remaining=127; running=2; done=98; failed=0; total=641; cuda0=9328MiB/90%; cuda1=7607MiB/77%; time=2026-08-15T22:15:13+08:00260815-22:15:14 (UTC+8)
162remaining=543; core_remaining=416; overflow_remaining=127; running=2; done=98; failed=0; total=641; cuda0=9328MiB/100%; cuda1=7607MiB/96%; time=2026-08-15T22:04:47+08:00260815-22:04:48 (UTC+8)
161remaining=544; core_remaining=417; overflow_remaining=127; running=2; done=97; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T21:54:21+08:00260815-21:54:22 (UTC+8)
160remaining=545; core_remaining=418; overflow_remaining=127; running=2; done=96; failed=0; total=641; cuda0=9326MiB/94%; cuda1=7607MiB/100%; time=2026-08-15T21:43:55+08:00260815-21:43:57 (UTC+8)
159remaining=546; core_remaining=419; overflow_remaining=127; running=2; done=95; failed=0; total=641; cuda0=9326MiB/92%; cuda1=7607MiB/97%; time=2026-08-15T21:33:29+08:00260815-21:33:31 (UTC+8)
158remaining=547; core_remaining=420; overflow_remaining=127; running=2; done=94; failed=0; total=641; cuda0=9326MiB/95%; cuda1=7607MiB/95%; time=2026-08-15T21:23:04+08:00260815-21:23:05 (UTC+8)
157remaining=547; core_remaining=420; overflow_remaining=127; running=2; done=94; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T21:12:37+08:00260815-21:12:39 (UTC+8)
156remaining=548; core_remaining=421; overflow_remaining=127; running=2; done=93; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/94%; time=2026-08-15T21:02:11+08:00260815-21:02:13 (UTC+8)
155remaining=549; core_remaining=422; overflow_remaining=127; running=2; done=92; failed=0; total=641; cuda0=9326MiB/84%; cuda1=7607MiB/100%; time=2026-08-15T20:51:46+08:00260815-20:51:47 (UTC+8)
154remaining=550; core_remaining=423; overflow_remaining=127; running=2; done=91; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T20:41:21+08:00260815-20:41:22 (UTC+8)
153remaining=551; core_remaining=424; overflow_remaining=127; running=2; done=90; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/94%; time=2026-08-15T20:30:54+08:00260815-20:30:56 (UTC+8)
152remaining=551; core_remaining=424; overflow_remaining=127; running=2; done=90; failed=0; total=641; cuda0=9326MiB/93%; cuda1=7607MiB/100%; time=2026-08-15T20:20:29+08:00260815-20:20:30 (UTC+8)
151remaining=552; core_remaining=425; overflow_remaining=127; running=2; done=89; failed=0; total=641; cuda0=9326MiB/88%; cuda1=7607MiB/100%; time=2026-08-15T20:10:03+08:00260815-20:10:04 (UTC+8)
150remaining=553; core_remaining=426; overflow_remaining=127; running=2; done=88; failed=0; total=641; cuda0=9326MiB/57%; cuda1=7607MiB/100%; time=2026-08-15T19:59:36+08:00260815-19:59:38 (UTC+8)
149remaining=554; core_remaining=427; overflow_remaining=127; running=2; done=87; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/89%; time=2026-08-15T19:49:11+08:00260815-19:49:12 (UTC+8)
148remaining=555; core_remaining=428; overflow_remaining=127; running=2; done=86; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/93%; time=2026-08-15T19:38:44+08:00260815-19:38:46 (UTC+8)
147remaining=555; core_remaining=428; overflow_remaining=127; running=2; done=86; failed=0; total=641; cuda0=9326MiB/91%; cuda1=7607MiB/96%; time=2026-08-15T19:28:18+08:00260815-19:28:20 (UTC+8)
146remaining=556; core_remaining=429; overflow_remaining=127; running=2; done=85; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T19:17:52+08:00260815-19:17:54 (UTC+8)
145remaining=557; core_remaining=430; overflow_remaining=127; running=2; done=84; failed=0; total=641; cuda0=9326MiB/89%; cuda1=7607MiB/96%; time=2026-08-15T19:07:27+08:00260815-19:07:27 (UTC+8)
144remaining=557; core_remaining=430; overflow_remaining=127; running=2; done=84; failed=0; total=641; cuda0=9326MiB/91%; cuda1=7607MiB/93%; time=2026-08-15T18:56:20+08:00260815-18:57:01 (UTC+8)
143remaining=559; core_remaining=432; overflow_remaining=127; running=2; done=82; failed=0; total=641; cuda0=9326MiB/87%; cuda1=7607MiB/100%; time=2026-08-15T18:46:19+08:00260815-18:46:35 (UTC+8)
142remaining=559; core_remaining=432; overflow_remaining=127; running=2; done=82; failed=0; total=641; cuda0=9326MiB/94%; cuda1=7607MiB/0%; time=2026-08-15T18:36:09+08:00260815-18:36:10 (UTC+8)
141remaining=560; core_remaining=433; overflow_remaining=127; running=2; done=81; failed=0; total=641; cuda0=9326MiB/56%; cuda1=7607MiB/100%; time=2026-08-15T18:25:41+08:00260815-18:25:43 (UTC+8)
140remaining=561; core_remaining=434; overflow_remaining=127; running=2; done=80; failed=0; total=641; cuda0=9326MiB/99%; cuda1=7607MiB/100%; time=2026-08-15T18:15:16+08:00260815-18:15:17 (UTC+8)
139remaining=561; core_remaining=434; overflow_remaining=127; running=2; done=80; failed=0; total=641; cuda0=9326MiB/91%; cuda1=7607MiB/100%; time=2026-08-15T18:04:50+08:00260815-18:04:51 (UTC+8)
138remaining=562; core_remaining=435; overflow_remaining=127; running=2; done=79; failed=0; total=641; cuda0=9326MiB/90%; cuda1=7607MiB/0%; time=2026-08-15T17:54:24+08:00260815-17:54:25 (UTC+8)
137remaining=563; core_remaining=436; overflow_remaining=127; running=2; done=78; failed=0; total=641; cuda0=9326MiB/95%; cuda1=7607MiB/89%; time=2026-08-15T17:43:58+08:00260815-17:43:59 (UTC+8)
136remaining=564; core_remaining=437; overflow_remaining=127; running=2; done=77; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T17:33:32+08:00260815-17:33:33 (UTC+8)
135remaining=565; core_remaining=438; overflow_remaining=127; running=2; done=76; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/92%; time=2026-08-15T17:23:06+08:00260815-17:23:07 (UTC+8)
134remaining=565; core_remaining=438; overflow_remaining=127; running=2; done=76; failed=0; total=641; cuda0=9326MiB/97%; cuda1=7607MiB/90%; time=2026-08-15T17:12:40+08:00260815-17:12:41 (UTC+8)
133remaining=566; core_remaining=439; overflow_remaining=127; running=2; done=75; failed=0; total=641; cuda0=9326MiB/93%; cuda1=7607MiB/100%; time=2026-08-15T17:02:13+08:00260815-17:02:15 (UTC+8)
132remaining=567; core_remaining=440; overflow_remaining=127; running=2; done=74; failed=0; total=641; cuda0=9326MiB/94%; cuda1=7607MiB/100%; time=2026-08-15T16:51:47+08:00260815-16:51:49 (UTC+8)
131remaining=568; core_remaining=441; overflow_remaining=127; running=2; done=73; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T16:41:22+08:00260815-16:41:23 (UTC+8)
130remaining=569; core_remaining=442; overflow_remaining=127; running=2; done=72; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/94%; time=2026-08-15T16:30:55+08:00260815-16:30:57 (UTC+8)
129remaining=569; core_remaining=442; overflow_remaining=127; running=2; done=72; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/95%; time=2026-08-15T16:20:29+08:00260815-16:20:31 (UTC+8)
128remaining=570; core_remaining=443; overflow_remaining=127; running=2; done=71; failed=0; total=641; cuda0=9326MiB/87%; cuda1=7607MiB/91%; time=2026-08-15T16:10:03+08:00260815-16:10:04 (UTC+8)
127remaining=571; core_remaining=444; overflow_remaining=127; running=2; done=70; failed=0; total=641; cuda0=9326MiB/96%; cuda1=7607MiB/92%; time=2026-08-15T15:59:37+08:00260815-15:59:38 (UTC+8)
126remaining=572; core_remaining=445; overflow_remaining=127; running=2; done=69; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/90%; time=2026-08-15T15:49:11+08:00260815-15:49:12 (UTC+8)
125remaining=573; core_remaining=446; overflow_remaining=127; running=2; done=68; failed=0; total=641; cuda0=9326MiB/91%; cuda1=7607MiB/100%; time=2026-08-15T15:38:44+08:00260815-15:38:46 (UTC+8)
124remaining=573; core_remaining=446; overflow_remaining=127; running=2; done=68; failed=0; total=641; cuda0=9326MiB/81%; cuda1=7607MiB/95%; time=2026-08-15T15:28:19+08:00260815-15:28:20 (UTC+8)
123remaining=574; core_remaining=447; overflow_remaining=127; running=2; done=67; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T15:17:53+08:00260815-15:17:54 (UTC+8)
122remaining=575; core_remaining=448; overflow_remaining=127; running=2; done=66; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/96%; time=2026-08-15T15:07:26+08:00260815-15:07:28 (UTC+8)
121remaining=575; core_remaining=448; overflow_remaining=127; running=2; done=66; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/91%; time=2026-08-15T14:57:00+08:00260815-14:57:01 (UTC+8)
120remaining=576; core_remaining=449; overflow_remaining=127; running=2; done=65; failed=0; total=641; cuda0=9326MiB/91%; cuda1=7607MiB/95%; time=2026-08-15T14:46:34+08:00260815-14:46:35 (UTC+8)
119remaining=577; core_remaining=450; overflow_remaining=127; running=2; done=64; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T14:36:07+08:00260815-14:36:09 (UTC+8)
118remaining=578; core_remaining=451; overflow_remaining=127; running=2; done=63; failed=0; total=641; cuda0=9326MiB/91%; cuda1=7607MiB/95%; time=2026-08-15T14:25:41+08:00260815-14:25:43 (UTC+8)
117remaining=579; core_remaining=452; overflow_remaining=127; running=2; done=62; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T14:15:15+08:00260815-14:15:16 (UTC+8)
116remaining=579; core_remaining=452; overflow_remaining=127; running=2; done=62; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/91%; time=2026-08-15T14:04:50+08:00260815-14:04:51 (UTC+8)
115remaining=581; core_remaining=454; overflow_remaining=127; running=2; done=60; failed=0; total=641; cuda0=9326MiB/100%; cuda1=7607MiB/99%; time=2026-08-15T13:43:47+08:00260815-13:43:49 (UTC+8)
114remaining=582; core_remaining=455; overflow_remaining=127; running=2; done=59; failed=0; total=641; cuda0=9323MiB/94%; cuda1=7607MiB/90%; time=2026-08-15T13:33:20+08:00260815-13:33:23 (UTC+8)
113remaining=582; core_remaining=455; overflow_remaining=127; running=2; done=59; failed=0; total=641; cuda0=9323MiB/96%; cuda1=7607MiB/100%; time=2026-08-15T13:22:54+08:00260815-13:22:55 (UTC+8)
112remaining=583; core_remaining=456; overflow_remaining=127; running=2; done=58; failed=0; total=641; cuda0=9323MiB/100%; cuda1=7607MiB/93%; time=2026-08-15T13:12:28+08:00260815-13:12:29 (UTC+8)
111remaining=584; core_remaining=457; overflow_remaining=127; running=2; done=57; failed=0; total=641; cuda0=7632MiB/87%; cuda1=7439MiB/39%; time=2026-08-15T12:51:27+08:00260815-12:51:28 (UTC+8)
110remaining=585; core_remaining=458; overflow_remaining=127; running=2; done=56; failed=0; total=641; cuda0=7632MiB/90%; cuda1=7607MiB/100%; time=2026-08-15T12:41:00+08:00260815-12:41:01 (UTC+8)
109remaining=586; core_remaining=459; overflow_remaining=127; running=2; done=55; failed=0; total=641; cuda0=7632MiB/66%; cuda1=7607MiB/100%; time=2026-08-15T12:30:34+08:00260815-12:30:35 (UTC+8)
108remaining=587; core_remaining=460; overflow_remaining=127; running=2; done=54; failed=0; total=641; cuda0=7632MiB/77%; cuda1=7607MiB/99%; time=2026-08-15T12:20:06+08:00260815-12:20:09 (UTC+8)
107remaining=587; core_remaining=460; overflow_remaining=127; running=2; done=54; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/95%; time=2026-08-15T12:09:41+08:00260815-12:09:43 (UTC+8)
106remaining=589; core_remaining=462; overflow_remaining=127; running=2; done=52; failed=0; total=641; cuda0=7632MiB/95%; cuda1=7607MiB/92%; time=2026-08-15T11:48:40+08:00260815-11:48:41 (UTC+8)
105remaining=589; core_remaining=462; overflow_remaining=127; running=2; done=52; failed=0; total=641; cuda0=7632MiB/96%; cuda1=7607MiB/100%; time=2026-08-15T11:38:14+08:00260815-11:38:15 (UTC+8)
104remaining=590; core_remaining=463; overflow_remaining=127; running=2; done=51; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/90%; time=2026-08-15T11:27:47+08:00260815-11:27:49 (UTC+8)
103remaining=591; core_remaining=464; overflow_remaining=127; running=2; done=50; failed=0; total=641; cuda0=7632MiB/92%; cuda1=7607MiB/100%; time=2026-08-15T11:17:21+08:00260815-11:17:23 (UTC+8)
102remaining=591; core_remaining=464; overflow_remaining=127; running=2; done=50; failed=0; total=641; cuda0=7632MiB/94%; cuda1=7607MiB/96%; time=2026-08-15T11:06:55+08:00260815-11:06:57 (UTC+8)
101remaining=592; core_remaining=465; overflow_remaining=127; running=2; done=49; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/91%; time=2026-08-15T10:56:29+08:00260815-10:56:30 (UTC+8)
100remaining=593; core_remaining=466; overflow_remaining=127; running=2; done=48; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/87%; time=2026-08-15T10:46:03+08:00260815-10:46:04 (UTC+8)
99remaining=594; core_remaining=467; overflow_remaining=127; running=2; done=47; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/89%; time=2026-08-15T10:35:37+08:00260815-10:35:38 (UTC+8)
98remaining=594; core_remaining=467; overflow_remaining=127; running=2; done=47; failed=0; total=641; cuda0=7632MiB/0%; cuda1=7607MiB/100%; time=2026-08-15T10:25:10+08:00260815-10:25:12 (UTC+8)
97remaining=595; core_remaining=468; overflow_remaining=127; running=2; done=46; failed=0; total=641; cuda0=7632MiB/83%; cuda1=7607MiB/100%; time=2026-08-15T10:14:44+08:00260815-10:14:46 (UTC+8)
96remaining=596; core_remaining=469; overflow_remaining=127; running=2; done=45; failed=0; total=641; cuda0=7632MiB/99%; cuda1=7607MiB/83%; time=2026-08-15T10:04:18+08:00260815-10:04:20 (UTC+8)
95remaining=596; core_remaining=469; overflow_remaining=127; running=2; done=45; failed=0; total=641; cuda0=7632MiB/96%; cuda1=7607MiB/100%; time=2026-08-15T09:53:52+08:00260815-09:53:54 (UTC+8)
94remaining=597; core_remaining=470; overflow_remaining=127; running=2; done=44; failed=0; total=641; cuda0=7632MiB/91%; cuda1=7607MiB/100%; time=2026-08-15T09:43:25+08:00260815-09:43:27 (UTC+8)
93remaining=598; core_remaining=471; overflow_remaining=127; running=2; done=43; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/91%; time=2026-08-15T09:33:00+08:00260815-09:33:01 (UTC+8)
92remaining=599; core_remaining=472; overflow_remaining=127; running=2; done=42; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T09:11:58+08:00260815-09:12:00 (UTC+8)
91remaining=600; core_remaining=473; overflow_remaining=127; running=2; done=41; failed=0; total=641; cuda0=7632MiB/96%; cuda1=7607MiB/100%; time=2026-08-15T09:01:32+08:00260815-09:01:33 (UTC+8)
90remaining=601; core_remaining=474; overflow_remaining=127; running=2; done=40; failed=0; total=641; cuda0=7632MiB/89%; cuda1=7607MiB/100%; time=2026-08-15T08:51:06+08:00260815-08:51:07 (UTC+8)
89remaining=601; core_remaining=474; overflow_remaining=127; running=2; done=40; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/98%; time=2026-08-15T08:40:40+08:00260815-08:40:41 (UTC+8)
88remaining=602; core_remaining=475; overflow_remaining=127; running=2; done=39; failed=0; total=641; cuda0=7632MiB/93%; cuda1=7607MiB/100%; time=2026-08-15T08:30:14+08:00260815-08:30:15 (UTC+8)
87remaining=603; core_remaining=476; overflow_remaining=127; running=2; done=38; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T08:19:48+08:00260815-08:19:49 (UTC+8)
86remaining=603; core_remaining=476; overflow_remaining=127; running=2; done=38; failed=0; total=641; cuda0=7632MiB/92%; cuda1=7607MiB/90%; time=2026-08-15T08:08:39+08:00260815-08:09:23 (UTC+8)
85remaining=604; core_remaining=477; overflow_remaining=127; running=2; done=37; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/95%; time=2026-08-15T07:58:38+08:00260815-07:58:57 (UTC+8)
84remaining=605; core_remaining=478; overflow_remaining=127; running=2; done=36; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T07:48:30+08:00260815-07:48:31 (UTC+8)
83remaining=606; core_remaining=479; overflow_remaining=127; running=2; done=35; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T07:27:27+08:00260815-07:27:29 (UTC+8)
82remaining=607; core_remaining=480; overflow_remaining=127; running=2; done=34; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T07:17:02+08:00260815-07:17:03 (UTC+8)
81remaining=608; core_remaining=481; overflow_remaining=127; running=2; done=33; failed=0; total=641; cuda0=7632MiB/92%; cuda1=7607MiB/100%; time=2026-08-15T07:05:39+08:00260815-07:06:37 (UTC+8)
80remaining=608; core_remaining=481; overflow_remaining=127; running=2; done=33; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/90%; time=2026-08-15T06:55:38+08:00260815-06:56:12 (UTC+8)
79remaining=609; core_remaining=482; overflow_remaining=127; running=2; done=32; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/0%; time=2026-08-15T06:45:36+08:00260815-06:45:45 (UTC+8)
78remaining=610; core_remaining=483; overflow_remaining=127; running=2; done=31; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/81%; time=2026-08-15T06:35:18+08:00260815-06:35:19 (UTC+8)
77remaining=610; core_remaining=483; overflow_remaining=127; running=2; done=31; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/100%; time=2026-08-15T06:24:52+08:00260815-06:24:53 (UTC+8)
76remaining=611; core_remaining=484; overflow_remaining=127; running=2; done=30; failed=0; total=641; cuda0=7632MiB/92%; cuda1=7607MiB/100%; time=2026-08-15T06:14:25+08:00260815-06:14:27 (UTC+8)
75remaining=612; core_remaining=485; overflow_remaining=127; running=2; done=29; failed=0; total=641; cuda0=7632MiB/92%; cuda1=7607MiB/91%; time=2026-08-15T06:04:00+08:00260815-06:04:00 (UTC+8)
74remaining=612; core_remaining=485; overflow_remaining=127; running=2; done=29; failed=0; total=641; cuda0=7632MiB/100%; cuda1=7607MiB/91%; time=2026-08-15T05:53:33+08:00260815-05:53:34 (UTC+8)
73remaining=613; core_remaining=486; overflow_remaining=127; running=2; done=28; failed=0; total=641; cuda0=7632MiB/100%; cuda1=12047MiB/100%; time=2026-08-15T05:43:07+08:00260815-05:43:08 (UTC+8)
72remaining=614; core_remaining=487; overflow_remaining=127; running=2; done=27; failed=0; total=641; cuda0=7632MiB/92%; cuda1=12047MiB/100%; time=2026-08-15T05:32:40+08:00260815-05:32:42 (UTC+8)
71remaining=615; core_remaining=488; overflow_remaining=127; running=2; done=26; failed=0; total=641; cuda0=7632MiB/100%; cuda1=12047MiB/100%; time=2026-08-15T05:11:39+08:00260815-05:11:41 (UTC+8)
70remaining=616; core_remaining=489; overflow_remaining=127; running=2; done=25; failed=0; total=641; cuda0=7632MiB/95%; cuda1=12047MiB/91%; time=2026-08-15T05:01:14+08:00260815-05:01:15 (UTC+8)
69remaining=617; core_remaining=490; overflow_remaining=127; running=2; done=24; failed=0; total=641; cuda0=7632MiB/100%; cuda1=12047MiB/100%; time=2026-08-15T04:50:47+08:00260815-04:50:48 (UTC+8)
68remaining=617; core_remaining=490; overflow_remaining=127; running=2; done=24; failed=0; total=641; cuda0=7632MiB/91%; cuda1=12047MiB/100%; time=2026-08-15T04:40:19+08:00260815-04:40:23 (UTC+8)
67remaining=618; core_remaining=491; overflow_remaining=127; running=2; done=23; failed=0; total=641; cuda0=7632MiB/90%; cuda1=12047MiB/100%; time=2026-08-15T04:29:55+08:00260815-04:29:56 (UTC+8)
66remaining=619; core_remaining=492; overflow_remaining=127; running=2; done=22; failed=0; total=641; cuda0=7632MiB/91%; cuda1=12049MiB/93%; time=2026-08-15T04:19:29+08:00260815-04:19:30 (UTC+8)
65remaining=619; core_remaining=492; overflow_remaining=127; running=2; done=22; failed=0; total=641; cuda0=7632MiB/100%; cuda1=12049MiB/93%; time=2026-08-15T04:09:03+08:00260815-04:09:04 (UTC+8)
64remaining=620; core_remaining=493; overflow_remaining=127; running=2; done=21; failed=0; total=641; cuda0=7632MiB/100%; cuda1=12049MiB/100%; time=2026-08-15T03:58:37+08:00260815-03:58:38 (UTC+8)
63remaining=621; core_remaining=494; overflow_remaining=127; running=2; done=20; failed=0; total=641; cuda0=7632MiB/92%; cuda1=12105MiB/100%; time=2026-08-15T03:48:10+08:00260815-03:48:12 (UTC+8)
62remaining=622; core_remaining=495; overflow_remaining=127; running=2; done=19; failed=0; total=641; cuda0=7632MiB/93%; cuda1=12105MiB/92%; time=2026-08-15T03:37:45+08:00260815-03:37:45 (UTC+8)
61remaining=622; core_remaining=495; overflow_remaining=127; running=2; done=19; failed=0; total=641; cuda0=7632MiB/90%; cuda1=12105MiB/83%; time=2026-08-15T03:27:19+08:00260815-03:27:20 (UTC+8)
60remaining=622; core_remaining=495; overflow_remaining=127; running=2; done=19; failed=0; total=641; cuda0=7632MiB/95%; cuda1=43078MiB/78%; time=2026-08-15T03:16:52+08:00260815-03:16:54 (UTC+8)
59remaining=623; core_remaining=496; overflow_remaining=127; running=2; done=18; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/83%; time=2026-08-15T03:06:25+08:00260815-03:06:27 (UTC+8)
58remaining=623; core_remaining=496; overflow_remaining=127; running=2; done=18; failed=0; total=641; cuda0=7632MiB/95%; cuda1=43078MiB/88%; time=2026-08-15T02:56:00+08:00260815-02:56:01 (UTC+8)
57remaining=623; core_remaining=496; overflow_remaining=127; running=2; done=18; failed=0; total=641; cuda0=7632MiB/90%; cuda1=43078MiB/99%; time=2026-08-15T02:45:33+08:00260815-02:45:35 (UTC+8)
56remaining=624; core_remaining=497; overflow_remaining=127; running=2; done=17; failed=0; total=641; cuda0=7632MiB/95%; cuda1=43078MiB/92%; time=2026-08-15T02:35:07+08:00260815-02:35:09 (UTC+8)
55remaining=624; core_remaining=497; overflow_remaining=127; running=2; done=17; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/91%; time=2026-08-15T02:24:41+08:00260815-02:24:42 (UTC+8)
54remaining=624; core_remaining=497; overflow_remaining=127; running=2; done=17; failed=0; total=641; cuda0=7632MiB/86%; cuda1=43078MiB/65%; time=2026-08-15T02:13:26+08:00260815-02:14:16 (UTC+8)
53remaining=625; core_remaining=498; overflow_remaining=127; running=2; done=16; failed=0; total=641; cuda0=7632MiB/0%; cuda1=43078MiB/76%; time=2026-08-15T02:03:25+08:00260815-02:03:49 (UTC+8)
52remaining=625; core_remaining=498; overflow_remaining=127; running=2; done=16; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/85%; time=2026-08-15T01:53:22+08:00260815-01:53:23 (UTC+8)
51remaining=625; core_remaining=498; overflow_remaining=127; running=2; done=16; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/97%; time=2026-08-15T01:42:55+08:00260815-01:42:57 (UTC+8)
50remaining=626; core_remaining=499; overflow_remaining=127; running=2; done=15; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/81%; time=2026-08-15T01:31:55+08:00260815-01:32:30 (UTC+8)
49remaining=626; core_remaining=499; overflow_remaining=127; running=2; done=15; failed=0; total=641; cuda0=7632MiB/96%; cuda1=43078MiB/84%; time=2026-08-15T01:21:54+08:00260815-01:22:05 (UTC+8)
48remaining=626; core_remaining=499; overflow_remaining=127; running=2; done=15; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/86%; time=2026-08-15T01:11:36+08:00260815-01:11:39 (UTC+8)
47remaining=627; core_remaining=500; overflow_remaining=127; running=2; done=14; failed=0; total=641; cuda0=7632MiB/92%; cuda1=43078MiB/91%; time=2026-08-15T01:01:11+08:00260815-01:01:12 (UTC+8)
46remaining=627; core_remaining=500; overflow_remaining=127; running=2; done=14; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/85%; time=2026-08-15T00:50:45+08:00260815-00:50:46 (UTC+8)
45remaining=627; core_remaining=500; overflow_remaining=127; running=2; done=14; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/79%; time=2026-08-15T00:40:18+08:00260815-00:40:20 (UTC+8)
44remaining=628; core_remaining=501; overflow_remaining=127; running=2; done=13; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/70%; time=2026-08-15T00:29:52+08:00260815-00:29:54 (UTC+8)
43remaining=628; core_remaining=501; overflow_remaining=127; running=2; done=13; failed=0; total=641; cuda0=7632MiB/100%; cuda1=43078MiB/77%; time=2026-08-15T00:19:27+08:00260815-00:19:28 (UTC+8)
42remaining=629; core_remaining=502; overflow_remaining=127; running=2; done=12; failed=0; total=641; cuda0=12976MiB/88%; cuda1=43078MiB/88%; time=2026-08-15T00:08:59+08:00260815-00:09:01 (UTC+8)
41remaining=629; core_remaining=502; overflow_remaining=127; running=2; done=12; failed=0; total=641; cuda0=12976MiB/100%; cuda1=43078MiB/83%; time=2026-08-14T23:57:55+08:00260814-23:58:35 (UTC+8)
40remaining=629; core_remaining=502; overflow_remaining=127; running=2; done=12; failed=0; total=641; cuda0=12976MiB/100%; cuda1=43078MiB/85%; time=2026-08-14T23:47:53+08:00260814-23:48:08 (UTC+8)
39remaining=630; core_remaining=503; overflow_remaining=127; running=2; done=11; failed=0; total=641; cuda0=12976MiB/92%; cuda1=43078MiB/82%; time=2026-08-14T23:37:39+08:00260814-23:37:42 (UTC+8)
38remaining=630; core_remaining=503; overflow_remaining=127; running=2; done=11; failed=0; total=641; cuda0=12976MiB/94%; cuda1=43078MiB/86%; time=2026-08-14T23:27:13+08:00260814-23:27:15 (UTC+8)
37remaining=630; core_remaining=503; overflow_remaining=127; running=2; done=11; failed=0; total=641; cuda0=12976MiB/90%; cuda1=43078MiB/79%; time=2026-08-14T23:16:46+08:00260814-23:16:48 (UTC+8)
36remaining=631; core_remaining=504; overflow_remaining=127; running=2; done=10; failed=0; total=641; cuda0=12976MiB/100%; cuda1=43078MiB/85%; time=2026-08-14T23:06:20+08:00260814-23:06:22 (UTC+8)
35remaining=631; core_remaining=504; overflow_remaining=127; running=2; done=10; failed=0; total=641; cuda0=12976MiB/94%; cuda1=43078MiB/79%; time=2026-08-14T22:55:54+08:00260814-22:55:55 (UTC+8)
34remaining=632; core_remaining=505; overflow_remaining=127; running=2; done=9; failed=0; total=641; cuda0=12976MiB/100%; cuda1=43292MiB/100%; time=2026-08-14T22:45:28+08:00260814-22:45:29 (UTC+8)
33remaining=633; core_remaining=506; overflow_remaining=127; running=2; done=8; failed=0; total=641; cuda0=12978MiB/100%; cuda1=43292MiB/100%; time=2026-08-14T22:35:02+08:00260814-22:35:03 (UTC+8)
32remaining=633; core_remaining=506; overflow_remaining=127; running=2; done=8; failed=0; total=641; cuda0=12978MiB/81%; cuda1=43292MiB/100%; time=2026-08-14T22:24:35+08:00260814-22:24:37 (UTC+8)
31remaining=633; core_remaining=506; overflow_remaining=127; running=2; done=8; failed=0; total=641; cuda0=12978MiB/91%; cuda1=43292MiB/100%; time=2026-08-14T22:14:09+08:00260814-22:14:10 (UTC+8)
30remaining=634; core_remaining=507; overflow_remaining=127; running=2; done=7; failed=0; total=641; cuda0=13034MiB/89%; cuda1=43292MiB/100%; time=2026-08-14T22:03:43+08:00260814-22:03:44 (UTC+8)
29remaining=634; core_remaining=507; overflow_remaining=127; running=2; done=7; failed=0; total=641; cuda0=13034MiB/100%; cuda1=43292MiB/95%; time=2026-08-14T21:53:18+08:00260814-21:53:18 (UTC+8)
28remaining=634; core_remaining=507; overflow_remaining=127; running=2; done=7; failed=0; total=641; cuda0=13034MiB/100%; cuda1=39908MiB/100%; time=2026-08-14T21:42:50+08:00260814-21:42:52 (UTC+8)
27remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=13248MiB/94%; cuda1=39150MiB/100%; time=2026-08-14T21:32:25+08:00260814-21:32:26 (UTC+8)
26remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=13248MiB/100%; cuda1=39150MiB/100%; time=2026-08-14T21:21:59+08:00260814-21:22:00 (UTC+8)
25remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=13248MiB/96%; cuda1=39150MiB/99%; time=2026-08-14T21:11:33+08:00260814-21:11:34 (UTC+8)
24remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=13248MiB/92%; cuda1=39150MiB/100%; time=2026-08-14T21:01:06+08:00260814-21:01:08 (UTC+8)
23remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=38215MiB/77%; cuda1=39150MiB/100%; time=2026-08-14T20:50:40+08:00260814-20:50:41 (UTC+8)
22remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=38215MiB/72%; cuda1=39150MiB/100%; time=2026-08-14T20:40:13+08:00260814-20:40:15 (UTC+8)
21remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=38215MiB/84%; cuda1=39150MiB/100%; time=2026-08-14T20:29:48+08:00260814-20:29:49 (UTC+8)
20remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=38215MiB/72%; cuda1=38612MiB/85%; time=2026-08-14T20:19:22+08:00260814-20:19:23 (UTC+8)
19remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=38215MiB/100%; cuda1=38612MiB/72%; time=2026-08-14T20:08:57+08:00260814-20:08:58 (UTC+8)
18remaining=636; core_remaining=509; overflow_remaining=127; running=2; done=5; failed=0; total=641; cuda0=38215MiB/78%; cuda1=38612MiB/86%; time=2026-08-14T19:58:31+08:00260814-19:58:32 (UTC+8)
17remaining=637; core_remaining=510; overflow_remaining=127; running=2; done=4; failed=0; total=641; cuda0=42173MiB/100%; cuda1=38612MiB/82%; time=2026-08-14T19:48:05+08:00260814-19:48:06 (UTC+8)
16remaining=637; core_remaining=510; overflow_remaining=127; running=2; done=4; failed=0; total=641; cuda0=40513MiB/100%; cuda1=38612MiB/82%; time=2026-08-14T19:37:39+08:00260814-19:37:40 (UTC+8)
15remaining=637; core_remaining=510; overflow_remaining=127; running=2; done=4; failed=0; total=641; cuda0=40513MiB/100%; cuda1=38612MiB/86%; time=2026-08-14T19:27:12+08:00260814-19:27:14 (UTC+8)
14remaining=637; core_remaining=510; overflow_remaining=127; running=2; done=4; failed=0; total=641; cuda0=40513MiB/100%; cuda1=38612MiB/88%; time=2026-08-14T19:15:51+08:00260814-19:16:47 (UTC+8)
13remaining=637; core_remaining=510; overflow_remaining=127; running=2; done=4; failed=0; total=641; cuda0=40513MiB/100%; cuda1=38612MiB/90%; time=2026-08-14T18:55:39+08:00260814-18:55:46 (UTC+8)
12remaining=637; core_remaining=510; overflow_remaining=127; running=2; done=4; failed=0; total=641; cuda0=40513MiB/100%; cuda1=38612MiB/79%; time=2026-08-14T18:45:17+08:00260814-18:45:19 (UTC+8)
11remaining=639; core_remaining=512; overflow_remaining=127; running=2; done=2; failed=0; total=641; cuda0=38559MiB/100%; cuda1=38738MiB/100%; time=2026-08-14T18:34:51+08:00260814-18:34:53 (UTC+8)
10remaining=639; core_remaining=512; overflow_remaining=127; running=2; done=2; failed=0; total=641; cuda0=38559MiB/97%; cuda1=38738MiB/90%; time=2026-08-14T18:24:25+08:00260814-18:24:27 (UTC+8)
9remaining=639; core_remaining=512; overflow_remaining=127; running=2; done=2; failed=0; total=641; cuda0=38559MiB/100%; cuda1=38738MiB/100%; time=2026-08-14T18:13:59+08:00260814-18:14:01 (UTC+8)
8remaining=639; core_remaining=512; overflow_remaining=127; running=2; done=2; failed=0; total=641; cuda0=38551MiB/100%; cuda1=38730MiB/100%; time=2026-08-14T18:03:33+08:00260814-18:03:34 (UTC+8)
7remaining=641; core_remaining=514; overflow_remaining=127; running=2; done=0; failed=0; total=641; cuda0=37427MiB/100%; cuda1=37606MiB/100%; time=2026-08-14T17:53:07+08:00260814-17:53:08 (UTC+8)
6remaining=641; core_remaining=514; overflow_remaining=127; running=2; done=0; failed=0; total=641; cuda0=37427MiB/100%; cuda1=37606MiB/100%; time=2026-08-14T17:42:40+08:00260814-17:42:42 (UTC+8)
5remaining=641; core_remaining=514; overflow_remaining=127; running=2; done=0; failed=0; total=641; cuda0=37427MiB/99%; cuda1=37606MiB/100%; time=2026-08-14T17:32:14+08:00260814-17:32:16 (UTC+8)
4remaining=641; core_remaining=514; overflow_remaining=127; running=2; done=0; failed=0; total=641; cuda0=37427MiB/100%; cuda1=37606MiB/100%; time=2026-08-14T17:21:48+08:00260814-17:21:49 (UTC+8)
3remaining=641; core_remaining=514; overflow_remaining=127; running=2; done=0; failed=0; total=641; cuda0=37427MiB/100%; cuda1=37606MiB/100%; time=2026-08-14T17:10:34+08:00260814-17:11:23 (UTC+8)
2remaining=641; running=2; done=0; failed=0; total=641; cuda0=37427MiB/96%; cuda1=37606MiB/100%; time=2026-08-14T17:09:52+08:00260814-17:10:39 (UTC+8)
1remaining=641; running=0; done=0; failed=0; total=641; cuda0=31016MiB/60%; cuda1=31195MiB/80%; time=2026-08-14T17:08:27+08:00260814-17:09:12 (UTC+8)
#05

TRACE-RCI260813four-datasetexperiment

▦ table
last update
260814-14:29:32
records
312
type
text
latest
tasks done=476/961 running=1 failed=0 blocked=0 pending=484 | results done=59/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:3930s | GPU cuda2:4643/24564MiB:26%;cuda3:1783/24564MiB:17%
ID数值接收时间
312tasks done=476/961 running=1 failed=0 blocked=0 pending=484 | results done=59/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:3930s | GPU cuda2:4643/24564MiB:26%;cuda3:1783/24564MiB:17%260814-14:29:32 (UTC+8)
311tasks done=476/961 running=1 failed=0 blocked=0 pending=484 | results done=59/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:3750s | GPU cuda2:4643/24564MiB:27%;cuda3:1783/24564MiB:17%260814-14:26:30 (UTC+8)
310tasks done=476/961 running=1 failed=0 blocked=0 pending=484 | results done=59/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:3569s | GPU cuda2:4643/24564MiB:26%;cuda3:1783/24564MiB:15%260814-14:23:30 (UTC+8)
309tasks done=476/961 running=1 failed=0 blocked=0 pending=484 | results done=59/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:3389s | GPU cuda2:4641/24564MiB:28%;cuda3:1783/24564MiB:18%260814-14:20:29 (UTC+8)
308tasks done=476/961 running=1 failed=0 blocked=0 pending=484 | results done=59/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:3208s | GPU cuda2:4641/24564MiB:26%;cuda3:1783/24564MiB:19%260814-14:17:29 (UTC+8)
307tasks done=476/961 running=1 failed=0 blocked=0 pending=484 | results done=59/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:3026s | GPU cuda2:4641/24564MiB:28%;cuda3:1783/24564MiB:16%260814-14:14:28 (UTC+8)
306tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:2845s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:5655s | GPU cuda2:4641/24564MiB:25%;cuda3:6413/24564MiB:93%260814-14:11:26 (UTC+8)
305tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:2665s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:5474s | GPU cuda2:4641/24564MiB:80%;cuda3:6413/24564MiB:77%260814-14:08:25 (UTC+8)
304tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:2484s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:5294s | GPU cuda2:4641/24564MiB:26%;cuda3:6413/24564MiB:77%260814-14:05:25 (UTC+8)
303tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:2304s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:5113s | GPU cuda2:4641/24564MiB:29%;cuda3:6413/24564MiB:93%260814-14:02:24 (UTC+8)
302tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:2123s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:4932s | GPU cuda2:4641/24564MiB:28%;cuda3:6413/24564MiB:100%260814-13:59:24 (UTC+8)
301tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:1942s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:4752s | GPU cuda2:4641/24564MiB:29%;cuda3:6413/24564MiB:89%260814-13:56:23 (UTC+8)
300tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:1570s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:4379s | GPU cuda2:4641/24564MiB:28%;cuda3:6413/24564MiB:88%260814-13:50:11 (UTC+8)
299tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:1389s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:4199s | GPU cuda2:4641/24564MiB:28%;cuda3:6413/24564MiB:95%260814-13:47:10 (UTC+8)
298tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:1208s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:4018s | GPU cuda2:4641/24564MiB:27%;cuda3:6413/24564MiB:94%260814-13:44:09 (UTC+8)
297tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:1027s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:3837s | GPU cuda2:4641/24564MiB:27%;cuda3:6413/24564MiB:93%260814-13:41:08 (UTC+8)
296tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:847s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:3657s | GPU cuda2:4641/24564MiB:27%;cuda3:6413/24564MiB:93%260814-13:38:08 (UTC+8)
295tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:666s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:3476s | GPU cuda2:4641/24564MiB:77%;cuda3:6411/24564MiB:89%260814-13:35:07 (UTC+8)
294tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:484s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:3293s | GPU cuda2:4641/24564MiB:26%;cuda3:6411/24564MiB:89%260814-13:32:07 (UTC+8)
293tasks done=471/961 running=2 failed=0 blocked=0 pending=488 | results done=58/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__citeseer__s888:303s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:3113s | GPU cuda2:4641/24564MiB:27%;cuda3:6411/24564MiB:81%260814-13:29:04 (UTC+8)
292tasks done=461/961 running=2 failed=0 blocked=0 pending=498 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__citeseer__s888:208s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:2552s | GPU cuda2:5405/24564MiB:100%;cuda3:6411/24564MiB:93%260814-13:19:43 (UTC+8)
291tasks done=461/961 running=2 failed=0 blocked=0 pending=498 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__citeseer__s888:27s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:2371s | GPU cuda2:5225/24564MiB:97%;cuda3:6411/24564MiB:93%260814-13:16:42 (UTC+8)
290tasks done=460/961 running=2 failed=0 blocked=0 pending=499 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__citeseer__s888:186s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:2190s | GPU cuda2:5405/24564MiB:96%;cuda3:6411/24564MiB:99%260814-13:13:41 (UTC+8)
289tasks done=460/961 running=2 failed=0 blocked=0 pending=499 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__citeseer__s888:6s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:2010s | GPU cuda2:401/24564MiB:0%;cuda3:6411/24564MiB:30%260814-13:10:41 (UTC+8)
288tasks done=458/961 running=2 failed=0 blocked=0 pending=501 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-512-b__llama31_4b__citeseer__s888:28s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:1829s | GPU cuda2:4805/24564MiB:97%;cuda3:6411/24564MiB:80%260814-13:07:40 (UTC+8)
287tasks done=457/961 running=2 failed=0 blocked=0 pending=502 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-512-a__llama31_4b__citeseer__s888:48s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:1647s | GPU cuda2:4865/24564MiB:97%;cuda3:6411/24564MiB:94%260814-13:04:40 (UTC+8)
286tasks done=454/961 running=2 failed=0 blocked=0 pending=505 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-256-a__llama31_4b__citeseer__s888:125s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:1466s | GPU cuda2:4565/24564MiB:95%;cuda3:6411/24564MiB:93%260814-13:01:38 (UTC+8)
285tasks done=452/961 running=2 failed=0 blocked=0 pending=507 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-128-b__llama31_4b__citeseer__s888:62s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:1286s | GPU cuda2:4565/24564MiB:95%;cuda3:6411/24564MiB:88%260814-12:58:37 (UTC+8)
284tasks done=449/961 running=2 failed=0 blocked=0 pending=510 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__citeseer__s888:491s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:1105s | GPU cuda2:5445/24564MiB:100%;cuda3:6411/24564MiB:88%260814-12:55:36 (UTC+8)
283tasks done=449/961 running=2 failed=0 blocked=0 pending=510 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__citeseer__s888:310s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:924s | GPU cuda2:5445/24564MiB:100%;cuda3:6411/24564MiB:81%260814-12:52:36 (UTC+8)
282tasks done=449/961 running=2 failed=0 blocked=0 pending=510 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__citeseer__s888:128s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:743s | GPU cuda2:5445/24564MiB:100%;cuda3:6411/24564MiB:76%260814-12:49:34 (UTC+8)
281tasks done=448/961 running=2 failed=0 blocked=0 pending=511 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__citeseer__s888:437s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:557s | GPU cuda2:5445/24564MiB:0%;cuda3:6411/24564MiB:81%260814-12:46:33 (UTC+8)
280tasks done=448/961 running=2 failed=0 blocked=0 pending=511 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__citeseer__s888:256s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:377s | GPU cuda2:5445/24564MiB:100%;cuda3:6411/24564MiB:91%260814-12:43:28 (UTC+8)
279tasks done=448/961 running=2 failed=0 blocked=0 pending=511 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__citeseer__s888:75s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:196s | GPU cuda2:5425/24564MiB:0%;cuda3:6391/24564MiB:94%260814-12:40:27 (UTC+8)
278tasks done=446/961 running=2 failed=0 blocked=0 pending=513 | results done=51/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-b__llama31_4b__citeseer__s888:230s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s666:15s | GPU cuda2:4865/24564MiB:96%;cuda3:6247/24564MiB:20%260814-12:37:27 (UTC+8)
277tasks done=443/961 running=2 failed=0 blocked=0 pending=516 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-b__llama31_4b__citeseer__s888:49s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:5580s | GPU cuda2:4865/24564MiB:100%;cuda3:6413/24564MiB:91%260814-12:34:26 (UTC+8)
276tasks done=442/961 running=2 failed=0 blocked=0 pending=517 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-a__llama31_4b__citeseer__s888:201s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:5399s | GPU cuda2:4865/24564MiB:97%;cuda3:6413/24564MiB:77%260814-12:31:25 (UTC+8)
275tasks done=442/961 running=2 failed=0 blocked=0 pending=517 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-a__llama31_4b__citeseer__s888:20s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:5218s | GPU cuda2:4845/24564MiB:97%;cuda3:6413/24564MiB:81%260814-12:28:24 (UTC+8)
274tasks done=440/961 running=2 failed=0 blocked=0 pending=519 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-b__llama31_4b__citeseer__s888:100s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:5038s | GPU cuda2:4605/24564MiB:96%;cuda3:6413/24564MiB:88%260814-12:25:23 (UTC+8)
273tasks done=439/961 running=2 failed=0 blocked=0 pending=520 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-a__llama31_4b__citeseer__s888:178s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:4856s | GPU cuda2:4605/24564MiB:97%;cuda3:6413/24564MiB:93%260814-12:22:23 (UTC+8)
272tasks done=437/961 running=2 failed=0 blocked=0 pending=522 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-b__llama31_4b__citeseer__s888:221s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:4674s | GPU cuda2:4565/24564MiB:95%;cuda3:6413/24564MiB:89%260814-12:19:21 (UTC+8)
271tasks done=437/961 running=2 failed=0 blocked=0 pending=522 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-b__llama31_4b__citeseer__s888:41s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:4494s | GPU cuda2:4565/24564MiB:0%;cuda3:6413/24564MiB:100%260814-12:16:19 (UTC+8)
270tasks done=436/961 running=2 failed=0 blocked=0 pending=523 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-a__llama31_4b__citeseer__s888:84s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:4313s | GPU cuda2:4565/24564MiB:0%;cuda3:6413/24564MiB:88%260814-12:13:19 (UTC+8)
269tasks done=434/961 running=2 failed=0 blocked=0 pending=525 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__citeseer__s888:305s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:4130s | GPU cuda2:5425/24564MiB:95%;cuda3:6413/24564MiB:81%260814-12:10:18 (UTC+8)
268tasks done=434/961 running=2 failed=0 blocked=0 pending=525 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__citeseer__s888:124s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:3949s | GPU cuda2:5425/24564MiB:57%;cuda3:6413/24564MiB:77%260814-12:07:15 (UTC+8)
267tasks done=433/961 running=2 failed=0 blocked=0 pending=526 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-a__llama31_4b__citeseer__s888:346s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:3768s | GPU cuda2:5425/24564MiB:100%;cuda3:6413/24564MiB:93%260814-12:04:14 (UTC+8)
266tasks done=433/961 running=2 failed=0 blocked=0 pending=526 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-a__llama31_4b__citeseer__s888:165s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:3588s | GPU cuda2:5425/24564MiB:100%;cuda3:6411/24564MiB:88%260814-12:01:13 (UTC+8)
265tasks done=431/961 running=2 failed=0 blocked=0 pending=528 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-b__llama31_4b__citeseer__s888:227s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:3407s | GPU cuda2:4885/24564MiB:96%;cuda3:6411/24564MiB:93%260814-11:58:13 (UTC+8)
264tasks done=431/961 running=2 failed=0 blocked=0 pending=528 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-b__llama31_4b__citeseer__s888:46s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:3226s | GPU cuda2:4865/24564MiB:96%;cuda3:6411/24564MiB:89%260814-11:55:12 (UTC+8)
263tasks done=430/961 running=2 failed=0 blocked=0 pending=529 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-a__llama31_4b__citeseer__s888:104s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:3045s | GPU cuda2:4885/24564MiB:100%;cuda3:6411/24564MiB:27%260814-11:52:11 (UTC+8)
262tasks done=428/961 running=2 failed=0 blocked=0 pending=531 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-128-b__llama31_4b__citeseer__s888:57s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:2863s | GPU cuda2:4565/24564MiB:95%;cuda3:6411/24564MiB:27%260814-11:49:10 (UTC+8)
261tasks done=427/961 running=2 failed=0 blocked=0 pending=532 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-128-a__llama31_4b__citeseer__s888:10s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:2683s | GPU cuda2:4427/24564MiB:13%;cuda3:6411/24564MiB:88%260814-11:46:08 (UTC+8)
260tasks done=425/961 running=2 failed=0 blocked=0 pending=534 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-b__llama31_4b__citeseer__s888:232s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:2502s | GPU cuda2:5445/24564MiB:0%;cuda3:6411/24564MiB:89%260814-11:43:07 (UTC+8)
259tasks done=425/961 running=2 failed=0 blocked=0 pending=534 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-b__llama31_4b__citeseer__s888:51s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:2321s | GPU cuda2:5425/24564MiB:95%;cuda3:6411/24564MiB:92%260814-11:40:07 (UTC+8)
258tasks done=424/961 running=2 failed=0 blocked=0 pending=535 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-a__llama31_4b__citeseer__s888:272s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:2140s | GPU cuda2:5445/24564MiB:95%;cuda3:6411/24564MiB:92%260814-11:37:06 (UTC+8)
257tasks done=424/961 running=2 failed=0 blocked=0 pending=535 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-a__llama31_4b__citeseer__s888:91s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:1960s | GPU cuda2:5445/24564MiB:100%;cuda3:6411/24564MiB:92%260814-11:34:05 (UTC+8)
256tasks done=422/961 running=2 failed=0 blocked=0 pending=537 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-b__llama31_4b__citeseer__s888:153s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:1779s | GPU cuda2:4865/24564MiB:97%;cuda3:6299/24564MiB:80%260814-11:31:05 (UTC+8)
255tasks done=421/961 running=2 failed=0 blocked=0 pending=538 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-a__llama31_4b__citeseer__s888:212s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:1598s | GPU cuda2:4865/24564MiB:97%;cuda3:6065/24564MiB:77%260814-11:28:04 (UTC+8)
254tasks done=421/961 running=2 failed=0 blocked=0 pending=538 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-a__llama31_4b__citeseer__s888:31s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:1418s | GPU cuda2:4865/24564MiB:96%;cuda3:6007/24564MiB:91%260814-11:25:03 (UTC+8)
253tasks done=419/961 running=2 failed=0 blocked=0 pending=540 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Random-256-b__llama31_4b__citeseer__s888:16s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:1237s | GPU cuda2:4549/24564MiB:0%;cuda3:6007/24564MiB:89%260814-11:22:03 (UTC+8)
252tasks done=417/961 running=2 failed=0 blocked=0 pending=542 | results done=50/120 | RUNNING cuda2:pair_audit:pair_audit__Random-128__llama31_4b__citeseer__s888:0s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:1056s | GPU cuda2:14/24564MiB:0%;cuda3:6007/24564MiB:89%260814-11:19:02 (UTC+8)
251tasks done=415/961 running=2 failed=0 blocked=0 pending=544 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Random-128-a__llama31_4b__citeseer__s888:81s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:876s | GPU cuda2:4565/24564MiB:98%;cuda3:6007/24564MiB:90%260814-11:16:01 (UTC+8)
250tasks done=413/961 running=2 failed=0 blocked=0 pending=546 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__citeseer__s888:302s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:695s | GPU cuda2:5445/24564MiB:100%;cuda3:5029/24564MiB:28%260814-11:13:00 (UTC+8)
249tasks done=413/961 running=2 failed=0 blocked=0 pending=546 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__citeseer__s888:121s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:514s | GPU cuda2:5833/24564MiB:94%;cuda3:5031/24564MiB:27%260814-11:10:00 (UTC+8)
248tasks done=412/961 running=2 failed=0 blocked=0 pending=547 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-a__llama31_4b__citeseer__s888:338s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:333s | GPU cuda2:5445/24564MiB:95%;cuda3:4641/24564MiB:26%260814-11:06:59 (UTC+8)
247tasks done=412/961 running=2 failed=0 blocked=0 pending=547 | results done=50/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-a__llama31_4b__citeseer__s888:157s;cuda3:duo_bank:duo_bank__llama31_4b__citeseer__s42:153s | GPU cuda2:5445/24564MiB:100%;cuda3:4621/24564MiB:27%260814-11:03:58 (UTC+8)
246tasks done=407/961 running=2 failed=0 blocked=0 pending=552 | results done=49/120 | RUNNING cuda2:trace_context:trace_bank__Full-256-b__llama31_4b__citeseer__s888:142s;cuda3:lingu_bank:lingu_bank__llama31_4b__citeseer__s888:340s | GPU cuda2:4605/24564MiB:95%;cuda3:5485/24564MiB:95%260814-11:00:57 (UTC+8)
245tasks done=406/961 running=2 failed=0 blocked=0 pending=553 | results done=49/120 | RUNNING cuda2:trace_context:trace_bank__Full-256-a__llama31_4b__citeseer__s888:124s;cuda3:lingu_bank:lingu_bank__llama31_4b__citeseer__s888:159s | GPU cuda2:4605/24564MiB:97%;cuda3:5465/24564MiB:95%260814-10:57:57 (UTC+8)
244tasks done=403/961 running=2 failed=0 blocked=0 pending=556 | results done=49/120 | RUNNING cuda2:trace_context:trace_bank__Full-128-b__llama31_4b__citeseer__s888:77s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s888:869s | GPU cuda2:4565/24564MiB:95%;cuda3:8451/24564MiB:100%260814-10:54:56 (UTC+8)
243tasks done=402/961 running=2 failed=0 blocked=0 pending=557 | results done=49/120 | RUNNING cuda2:trace_context:trace_bank__Full-128-a__llama31_4b__citeseer__s888:26s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s888:688s | GPU cuda2:4565/24564MiB:76%;cuda3:8449/24564MiB:100%260814-10:51:56 (UTC+8)
242tasks done=400/961 running=2 failed=0 blocked=0 pending=559 | results done=49/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-256-b__llama31_4b__citeseer__s888:13s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s888:507s | GPU cuda2:4449/24564MiB:13%;cuda3:8449/24564MiB:100%260814-10:48:54 (UTC+8)
241tasks done=398/961 running=2 failed=0 blocked=0 pending=561 | results done=49/120 | RUNNING cuda2:pair_audit:pair_audit__Full-512__llama31_4b__citeseer__s888:0s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s888:326s | GPU cuda2:14/24564MiB:0%;cuda3:8449/24564MiB:100%260814-10:45:53 (UTC+8)
240tasks done=397/961 running=2 failed=0 blocked=0 pending=562 | results done=49/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-b__llama31_4b__citeseer__s888:58s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s888:146s | GPU cuda2:4865/24564MiB:100%;cuda3:8387/24564MiB:100%260814-10:42:53 (UTC+8)
239tasks done=393/961 running=2 failed=0 blocked=0 pending=566 | results done=48/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-a__llama31_4b__citeseer__s888:113s;cuda3:lingu_bank:lingu_bank__llama31_4b__citeseer__s666:333s | GPU cuda2:4865/24564MiB:100%;cuda3:5485/24564MiB:100%260814-10:39:52 (UTC+8)
238tasks done=392/961 running=2 failed=0 blocked=0 pending=567 | results done=48/120 | RUNNING cuda2:trace_context:trace_bank__Self__llama31_4b__citeseer__s888:42s;cuda3:lingu_bank:lingu_bank__llama31_4b__citeseer__s666:152s | GPU cuda2:4565/24564MiB:97%;cuda3:5465/24564MiB:100%260814-10:36:50 (UTC+8)
237tasks done=390/961 running=2 failed=0 blocked=0 pending=569 | results done=48/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__citeseer__s888:541s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s666:864s | GPU cuda2:7949/24564MiB:100%;cuda3:8449/24564MiB:100%260814-10:33:49 (UTC+8)
236tasks done=390/961 running=2 failed=0 blocked=0 pending=569 | results done=48/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__citeseer__s888:360s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s666:684s | GPU cuda2:7949/24564MiB:100%;cuda3:8449/24564MiB:100%260814-10:30:48 (UTC+8)
235tasks done=390/961 running=2 failed=0 blocked=0 pending=569 | results done=48/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__citeseer__s888:179s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s666:503s | GPU cuda2:7947/24564MiB:100%;cuda3:8449/24564MiB:64%260814-10:27:48 (UTC+8)
234tasks done=389/961 running=2 failed=0 blocked=0 pending=570 | results done=47/120 | RUNNING cuda2:rci_downstream:run__RCI_Unconditioned__llama31_4b__citeseer__s666__gcn3:14s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s666:322s | GPU cuda2:651/24564MiB:19%;cuda3:8449/24564MiB:80%260814-10:24:47 (UTC+8)
233tasks done=380/961 running=2 failed=0 blocked=0 pending=579 | results done=41/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__citeseer__s666:280s;cuda3:sft:sft__lingu__llama31_4b__citeseer__s666:141s | GPU cuda2:5405/24564MiB:96%;cuda3:8449/24564MiB:99%260814-10:21:46 (UTC+8)
232tasks done=377/961 running=2 failed=0 blocked=0 pending=582 | results done=38/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__citeseer__s666:99s;cuda3:rci_downstream:run__RCI_Static__llama31_4b__citeseer__s42__gcn3:1s | GPU cuda2:5365/24564MiB:100%;cuda3:14/24564MiB:0%260814-10:18:45 (UTC+8)
231tasks done=369/961 running=2 failed=0 blocked=0 pending=590 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__citeseer__s666:260s;cuda3:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__citeseer__s42:233s | GPU cuda2:5405/24564MiB:95%;cuda3:5405/24564MiB:96%260814-10:15:44 (UTC+8)
230tasks done=369/961 running=2 failed=0 blocked=0 pending=590 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__citeseer__s666:79s;cuda3:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__citeseer__s42:52s | GPU cuda2:5345/24564MiB:97%;cuda3:5345/24564MiB:100%260814-10:12:43 (UTC+8)
229tasks done=366/961 running=2 failed=0 blocked=0 pending=593 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-512-b__llama31_4b__citeseer__s666:102s;cuda3:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__citeseer__s42:203s | GPU cuda2:4865/24564MiB:97%;cuda3:5405/24564MiB:100%260814-10:09:42 (UTC+8)
228tasks done=365/961 running=2 failed=0 blocked=0 pending=594 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-512-a__llama31_4b__citeseer__s666:124s;cuda3:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__citeseer__s42:22s | GPU cuda2:4865/24564MiB:100%;cuda3:5225/24564MiB:96%260814-10:06:41 (UTC+8)
227tasks done=361/961 running=2 failed=0 blocked=0 pending=598 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-256-b__llama31_4b__citeseer__s666:74s;cuda3:trace_context:trace_bank__HopQuota-512-b__llama31_4b__citeseer__s42:41s | GPU cuda2:4565/24564MiB:95%;cuda3:4865/24564MiB:97%260814-10:03:40 (UTC+8)
226tasks done=359/961 running=2 failed=0 blocked=0 pending=600 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-256-a__llama31_4b__citeseer__s666:21s;cuda3:trace_context:trace_bank__HopQuota-512-a__llama31_4b__citeseer__s42:58s | GPU cuda2:4565/24564MiB:97%;cuda3:4865/24564MiB:97%260814-10:00:40 (UTC+8)
225tasks done=354/961 running=2 failed=0 blocked=0 pending=605 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-128-a__llama31_4b__citeseer__s666:74s;cuda3:trace_context:trace_bank__HopQuota-256-b__llama31_4b__citeseer__s42:4s | GPU cuda2:4565/24564MiB:95%;cuda3:401/24564MiB:0%260814-09:57:39 (UTC+8)
224tasks done=349/961 running=2 failed=0 blocked=0 pending=610 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__citeseer__s666:387s;cuda3:trace_context:trace_bank__HopQuota-128-b__llama31_4b__citeseer__s42:65s | GPU cuda2:5445/24564MiB:100%;cuda3:4565/24564MiB:95%260814-09:54:38 (UTC+8)
223tasks done=346/961 running=2 failed=0 blocked=0 pending=613 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__citeseer__s666:206s;cuda3:trace_context:trace_bank__Diverse-1024-b__llama31_4b__citeseer__s42:483s | GPU cuda2:5445/24564MiB:96%;cuda3:2441/24564MiB:0%260814-09:51:38 (UTC+8)
222tasks done=346/961 running=2 failed=0 blocked=0 pending=613 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__citeseer__s666:25s;cuda3:trace_context:trace_bank__Diverse-1024-b__llama31_4b__citeseer__s42:303s | GPU cuda2:5405/24564MiB:95%;cuda3:5445/24564MiB:100%260814-09:48:37 (UTC+8)
221tasks done=345/961 running=2 failed=0 blocked=0 pending=614 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__citeseer__s666:336s;cuda3:trace_context:trace_bank__Diverse-1024-b__llama31_4b__citeseer__s42:122s | GPU cuda2:5445/24564MiB:95%;cuda3:5445/24564MiB:100%260814-09:45:36 (UTC+8)
220tasks done=344/961 running=2 failed=0 blocked=0 pending=615 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__citeseer__s666:155s;cuda3:trace_context:trace_bank__Diverse-1024-a__llama31_4b__citeseer__s42:472s | GPU cuda2:6041/24564MiB:95%;cuda3:6293/24564MiB:15%260814-09:42:36 (UTC+8)
219tasks done=342/961 running=2 failed=0 blocked=0 pending=617 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-b__llama31_4b__citeseer__s666:306s;cuda3:trace_context:trace_bank__Diverse-1024-a__llama31_4b__citeseer__s42:292s | GPU cuda2:5461/24564MiB:0%;cuda3:6293/24564MiB:96%260814-09:39:35 (UTC+8)
218tasks done=342/961 running=2 failed=0 blocked=0 pending=617 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-b__llama31_4b__citeseer__s666:125s;cuda3:trace_context:trace_bank__Diverse-1024-a__llama31_4b__citeseer__s42:111s | GPU cuda2:5461/24564MiB:98%;cuda3:6273/24564MiB:96%260814-09:36:34 (UTC+8)
217tasks done=339/961 running=2 failed=0 blocked=0 pending=620 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-a__llama31_4b__citeseer__s666:278s;cuda3:trace_context:trace_bank__Diverse-512-b__llama31_4b__citeseer__s42:285s | GPU cuda2:5461/24564MiB:97%;cuda3:5713/24564MiB:15%260814-09:33:33 (UTC+8)
216tasks done=339/961 running=2 failed=0 blocked=0 pending=620 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-a__llama31_4b__citeseer__s666:97s;cuda3:trace_context:trace_bank__Diverse-512-b__llama31_4b__citeseer__s42:105s | GPU cuda2:5461/24564MiB:97%;cuda3:5713/24564MiB:100%260814-09:30:32 (UTC+8)
215tasks done=336/961 running=2 failed=0 blocked=0 pending=623 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-b__llama31_4b__citeseer__s666:175s;cuda3:trace_context:trace_bank__Diverse-512-a__llama31_4b__citeseer__s42:276s | GPU cuda2:5201/24564MiB:97%;cuda3:5713/24564MiB:97%260814-09:27:32 (UTC+8)
214tasks done=335/961 running=2 failed=0 blocked=0 pending=624 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-a__llama31_4b__citeseer__s666:250s;cuda3:trace_context:trace_bank__Diverse-512-a__llama31_4b__citeseer__s42:95s | GPU cuda2:5201/24564MiB:97%;cuda3:5713/24564MiB:98%260814-09:24:31 (UTC+8)
213tasks done=333/961 running=2 failed=0 blocked=0 pending=626 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-a__llama31_4b__citeseer__s666:69s;cuda3:trace_context:trace_bank__Diverse-256-b__llama31_4b__citeseer__s42:188s | GPU cuda2:5201/24564MiB:95%;cuda3:5453/24564MiB:98%260814-09:21:30 (UTC+8)
212tasks done=331/961 running=2 failed=0 blocked=0 pending=628 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-b__llama31_4b__citeseer__s666:115s;cuda3:trace_context:trace_bank__Diverse-256-b__llama31_4b__citeseer__s42:8s | GPU cuda2:5161/24564MiB:95%;cuda3:5261/24564MiB:27%260814-09:18:29 (UTC+8)
211tasks done=329/961 running=2 failed=0 blocked=0 pending=630 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-a__llama31_4b__citeseer__s666:161s;cuda3:trace_context:trace_bank__Diverse-256-a__llama31_4b__citeseer__s42:101s | GPU cuda2:5161/24564MiB:0%;cuda3:5453/24564MiB:15%260814-09:15:29 (UTC+8)
210tasks done=325/961 running=2 failed=0 blocked=0 pending=634 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__citeseer__s666:382s;cuda3:trace_context:trace_bank__Diverse-128-b__llama31_4b__citeseer__s42:157s | GPU cuda2:6021/24564MiB:100%;cuda3:5413/24564MiB:15%260814-09:12:28 (UTC+8)
209tasks done=324/961 running=2 failed=0 blocked=0 pending=635 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__citeseer__s666:202s;cuda3:trace_context:trace_bank__Diverse-128-a__llama31_4b__citeseer__s42:215s | GPU cuda2:6021/24564MiB:88%;cuda3:5413/24564MiB:99%260814-09:09:27 (UTC+8)
208tasks done=324/961 running=2 failed=0 blocked=0 pending=635 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__citeseer__s666:21s;cuda3:trace_context:trace_bank__Diverse-128-a__llama31_4b__citeseer__s42:34s | GPU cuda2:6001/24564MiB:95%;cuda3:5413/24564MiB:15%260814-09:06:27 (UTC+8)
207tasks done=321/961 running=2 failed=0 blocked=0 pending=638 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-a__llama31_4b__citeseer__s666:243s;cuda3:trace_context:trace_bank__Semantic-1024-b__llama31_4b__citeseer__s42:299s | GPU cuda2:6021/24564MiB:100%;cuda3:6273/24564MiB:100%260814-09:03:26 (UTC+8)
206tasks done=321/961 running=2 failed=0 blocked=0 pending=638 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-a__llama31_4b__citeseer__s666:63s;cuda3:trace_context:trace_bank__Semantic-1024-b__llama31_4b__citeseer__s42:118s | GPU cuda2:6021/24564MiB:57%;cuda3:6273/24564MiB:97%260814-09:00:25 (UTC+8)
205tasks done=318/961 running=2 failed=0 blocked=0 pending=641 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-b__llama31_4b__citeseer__s666:124s;cuda3:trace_context:trace_bank__Semantic-1024-a__llama31_4b__citeseer__s42:381s | GPU cuda2:5481/24564MiB:74%;cuda3:6273/24564MiB:100%260814-08:57:24 (UTC+8)
204tasks done=317/961 running=2 failed=0 blocked=0 pending=642 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-a__llama31_4b__citeseer__s666:183s;cuda3:trace_context:trace_bank__Semantic-1024-a__llama31_4b__citeseer__s42:200s | GPU cuda2:5481/24564MiB:97%;cuda3:6273/24564MiB:99%260814-08:54:23 (UTC+8)
203tasks done=317/961 running=2 failed=0 blocked=0 pending=642 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-a__llama31_4b__citeseer__s666:3s;cuda3:trace_context:trace_bank__Semantic-1024-a__llama31_4b__citeseer__s42:19s | GPU cuda2:997/24564MiB:0%;cuda3:6253/24564MiB:100%260814-08:51:22 (UTC+8)
202tasks done=312/961 running=2 failed=0 blocked=0 pending=647 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-128-a__llama31_4b__citeseer__s666:90s;cuda3:trace_context:trace_bank__Semantic-512-b__llama31_4b__citeseer__s42:102s | GPU cuda2:5161/24564MiB:95%;cuda3:5733/24564MiB:97%260814-08:48:21 (UTC+8)
201tasks done=309/961 running=2 failed=0 blocked=0 pending=650 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-b__llama31_4b__citeseer__s666:312s;cuda3:trace_context:trace_bank__Semantic-512-a__llama31_4b__citeseer__s42:183s | GPU cuda2:6041/24564MiB:100%;cuda3:5733/24564MiB:15%260814-08:45:21 (UTC+8)
200tasks done=309/961 running=2 failed=0 blocked=0 pending=650 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-b__llama31_4b__citeseer__s666:131s;cuda3:trace_context:trace_bank__Semantic-512-a__llama31_4b__citeseer__s42:2s | GPU cuda2:6021/24564MiB:95%;cuda3:1249/24564MiB:18%260814-08:42:20 (UTC+8)
199tasks done=305/961 running=2 failed=0 blocked=0 pending=654 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-a__llama31_4b__citeseer__s666:351s;cuda3:trace_context:trace_bank__Semantic-128-a__llama31_4b__citeseer__s42:109s | GPU cuda2:6041/24564MiB:100%;cuda3:5413/24564MiB:99%260814-08:39:19 (UTC+8)
198tasks done=303/961 running=2 failed=0 blocked=0 pending=656 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-a__llama31_4b__citeseer__s666:171s;cuda3:trace_context:trace_bank__Random-1024-b__llama31_4b__citeseer__s42:371s | GPU cuda2:6041/24564MiB:100%;cuda3:6273/24564MiB:100%260814-08:36:18 (UTC+8)
197tasks done=301/961 running=2 failed=0 blocked=0 pending=658 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-b__llama31_4b__citeseer__s666:231s;cuda3:trace_context:trace_bank__Random-1024-b__llama31_4b__citeseer__s42:190s | GPU cuda2:5481/24564MiB:97%;cuda3:6273/24564MiB:96%260814-08:33:18 (UTC+8)
196tasks done=301/961 running=2 failed=0 blocked=0 pending=658 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-b__llama31_4b__citeseer__s666:50s;cuda3:trace_context:trace_bank__Random-1024-b__llama31_4b__citeseer__s42:10s | GPU cuda2:5461/24564MiB:97%;cuda3:5277/24564MiB:28%260814-08:30:17 (UTC+8)
195tasks done=299/961 running=2 failed=0 blocked=0 pending=660 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-a__llama31_4b__citeseer__s666:107s;cuda3:trace_context:trace_bank__Random-1024-a__llama31_4b__citeseer__s42:269s | GPU cuda2:5461/24564MiB:96%;cuda3:6273/24564MiB:100%260814-08:27:17 (UTC+8)
194tasks done=297/961 running=2 failed=0 blocked=0 pending=662 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-256-b__llama31_4b__citeseer__s666:91s;cuda3:trace_context:trace_bank__Random-1024-a__llama31_4b__citeseer__s42:89s | GPU cuda2:5201/24564MiB:97%;cuda3:6273/24564MiB:96%260814-08:24:16 (UTC+8)
193tasks done=294/961 running=2 failed=0 blocked=0 pending=665 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-256-a__llama31_4b__citeseer__s666:73s;cuda3:trace_context:trace_bank__Random-512-b__llama31_4b__citeseer__s42:170s | GPU cuda2:5181/24564MiB:95%;cuda3:5713/24564MiB:98%260814-08:21:15 (UTC+8)
192tasks done=291/961 running=2 failed=0 blocked=0 pending=668 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Random-128-b__llama31_4b__citeseer__s666:26s;cuda3:trace_context:trace_bank__Random-512-a__llama31_4b__citeseer__s42:249s | GPU cuda2:5161/24564MiB:28%;cuda3:5733/24564MiB:100%260814-08:18:15 (UTC+8)
191tasks done=288/961 running=2 failed=0 blocked=0 pending=671 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__citeseer__s666:380s;cuda3:trace_context:trace_bank__Random-512-a__llama31_4b__citeseer__s42:68s | GPU cuda2:6041/24564MiB:95%;cuda3:5713/24564MiB:98%260814-08:15:14 (UTC+8)
190tasks done=286/961 running=2 failed=0 blocked=0 pending=673 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__citeseer__s666:200s;cuda3:trace_context:trace_bank__Random-256-b__llama31_4b__citeseer__s42:65s | GPU cuda2:6041/24564MiB:95%;cuda3:5453/24564MiB:98%260814-08:12:14 (UTC+8)
189tasks done=285/961 running=2 failed=0 blocked=0 pending=674 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__citeseer__s666:19s;cuda3:trace_context:trace_bank__Random-256-a__llama31_4b__citeseer__s42:60s | GPU cuda2:6001/24564MiB:94%;cuda3:5453/24564MiB:98%260814-08:09:13 (UTC+8)
188tasks done=282/961 running=2 failed=0 blocked=0 pending=677 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-a__llama31_4b__citeseer__s666:240s;cuda3:trace_context:trace_bank__Random-128-b__llama31_4b__citeseer__s42:21s | GPU cuda2:6041/24564MiB:100%;cuda3:5413/24564MiB:74%260814-08:06:12 (UTC+8)
187tasks done=279/961 running=2 failed=0 blocked=0 pending=680 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-a__llama31_4b__citeseer__s666:59s;cuda3:trace_context:trace_bank__Full-1024-b__llama31_4b__citeseer__s42:424s | GPU cuda2:6021/24564MiB:100%;cuda3:6293/24564MiB:79%260814-08:03:11 (UTC+8)
186tasks done=277/961 running=2 failed=0 blocked=0 pending=682 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-256-b__llama31_4b__citeseer__s666:44s;cuda3:trace_context:trace_bank__Full-1024-b__llama31_4b__citeseer__s42:243s | GPU cuda2:5201/24564MiB:97%;cuda3:6293/24564MiB:96%260814-08:00:11 (UTC+8)
185tasks done=276/961 running=2 failed=0 blocked=0 pending=683 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-256-a__llama31_4b__citeseer__s666:25s;cuda3:trace_context:trace_bank__Full-1024-b__llama31_4b__citeseer__s42:62s | GPU cuda2:5181/24564MiB:97%;cuda3:6273/24564MiB:100%260814-07:57:10 (UTC+8)
184tasks done=272/961 running=2 failed=0 blocked=0 pending=687 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-128-a__llama31_4b__citeseer__s666:109s;cuda3:trace_context:trace_bank__Full-1024-a__llama31_4b__citeseer__s42:323s | GPU cuda2:5161/24564MiB:95%;cuda3:6293/24564MiB:96%260814-07:54:09 (UTC+8)
183tasks done=270/961 running=2 failed=0 blocked=0 pending=689 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-256-b__llama31_4b__citeseer__s666:96s;cuda3:trace_context:trace_bank__Full-1024-a__llama31_4b__citeseer__s42:143s | GPU cuda2:5201/24564MiB:99%;cuda3:6293/24564MiB:96%260814-07:51:08 (UTC+8)
182tasks done=267/961 running=2 failed=0 blocked=0 pending=692 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-256-a__llama31_4b__citeseer__s666:81s;cuda3:trace_context:trace_bank__Full-256-b__llama31_4b__citeseer__s42:139s | GPU cuda2:5201/24564MiB:97%;cuda3:5453/24564MiB:98%260814-07:48:08 (UTC+8)
181tasks done=264/961 running=2 failed=0 blocked=0 pending=695 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-b__llama31_4b__citeseer__s666:140s;cuda3:trace_context:trace_bank__Full-256-a__llama31_4b__citeseer__s42:134s | GPU cuda2:5461/24564MiB:96%;cuda3:5453/24564MiB:98%260814-07:45:07 (UTC+8)
180tasks done=261/961 running=2 failed=0 blocked=0 pending=698 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-a__llama31_4b__citeseer__s666:197s;cuda3:trace_context:trace_bank__Full-128-b__llama31_4b__citeseer__s42:96s | GPU cuda2:5461/24564MiB:100%;cuda3:5413/24564MiB:99%260814-07:42:06 (UTC+8)
179tasks done=260/961 running=2 failed=0 blocked=0 pending=699 | results done=34/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-a__llama31_4b__citeseer__s666:16s;cuda3:trace_context:trace_bank__Full-128-a__llama31_4b__citeseer__s42:56s | GPU cuda2:5161/24564MiB:1%;cuda3:5413/24564MiB:67%260814-07:39:05 (UTC+8)
178tasks done=256/961 running=2 failed=0 blocked=0 pending=703 | results done=34/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__citeseer__s666:621s;cuda3:trace_context:trace_bank__Semantic-256-b__llama31_4b__citeseer__s42:56s | GPU cuda2:8543/24564MiB:95%;cuda3:5453/24564MiB:15%260814-07:36:05 (UTC+8)
177tasks done=255/961 running=2 failed=0 blocked=0 pending=704 | results done=34/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__citeseer__s666:440s;cuda3:trace_context:trace_bank__Semantic-256-a__llama31_4b__citeseer__s42:53s | GPU cuda2:8543/24564MiB:100%;cuda3:5453/24564MiB:98%260814-07:33:04 (UTC+8)
176tasks done=253/961 running=2 failed=0 blocked=0 pending=706 | results done=34/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__citeseer__s666:260s;cuda3:trace_context:trace_bank__Full-512-b__llama31_4b__citeseer__s42:133s | GPU cuda2:8543/24564MiB:100%;cuda3:5713/24564MiB:98%260814-07:30:03 (UTC+8)
175tasks done=252/961 running=2 failed=0 blocked=0 pending=707 | results done=34/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__citeseer__s666:79s;cuda3:trace_context:trace_bank__Full-512-a__llama31_4b__citeseer__s42:275s | GPU cuda2:8543/24564MiB:57%;cuda3:5713/24564MiB:98%260814-07:27:02 (UTC+8)
174tasks done=249/961 running=2 failed=0 blocked=0 pending=710 | results done=33/120 | RUNNING cuda2:lingu_bank:lingu_bank__llama31_4b__citeseer__s42:277s;cuda3:trace_context:trace_bank__Full-512-a__llama31_4b__citeseer__s42:94s | GPU cuda2:6081/24564MiB:100%;cuda3:7616/24564MiB:100%260814-07:24:01 (UTC+8)
173tasks done=248/961 running=2 failed=0 blocked=0 pending=711 | results done=33/120 | RUNNING cuda2:lingu_bank:lingu_bank__llama31_4b__citeseer__s42:97s;cuda3:trace_context:trace_bank__Self__llama31_4b__citeseer__s42:69s | GPU cuda2:6061/24564MiB:100%;cuda3:7316/24564MiB:100%260814-07:21:01 (UTC+8)
172tasks done=246/961 running=2 failed=0 blocked=0 pending=713 | results done=33/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__citeseer__s42:825s;cuda3:sft:sft__trace__llama31_4b__citeseer__s42:830s | GPU cuda2:9047/24564MiB:98%;cuda3:10698/24564MiB:100%260814-07:18:00 (UTC+8)
171tasks done=246/961 running=2 failed=0 blocked=0 pending=713 | results done=33/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__citeseer__s42:645s;cuda3:sft:sft__trace__llama31_4b__citeseer__s42:650s | GPU cuda2:9047/24564MiB:79%;cuda3:10698/24564MiB:100%260814-07:14:59 (UTC+8)
170tasks done=246/961 running=2 failed=0 blocked=0 pending=713 | results done=33/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__citeseer__s42:464s;cuda3:sft:sft__trace__llama31_4b__citeseer__s42:469s | GPU cuda2:9045/24564MiB:99%;cuda3:10698/24564MiB:100%260814-07:11:59 (UTC+8)
169tasks done=246/961 running=2 failed=0 blocked=0 pending=713 | results done=33/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__citeseer__s42:283s;cuda3:sft:sft__trace__llama31_4b__citeseer__s42:288s | GPU cuda2:9005/24564MiB:94%;cuda3:10698/24564MiB:100%260814-07:08:58 (UTC+8)
168tasks done=246/961 running=2 failed=0 blocked=0 pending=713 | results done=33/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__citeseer__s42:103s;cuda3:sft:sft__trace__llama31_4b__citeseer__s42:108s | GPU cuda2:8985/24564MiB:100%;cuda3:10698/24564MiB:100%260814-07:05:57 (UTC+8)
167tasks done=236/961 running=1 failed=0 blocked=0 pending=724 | results done=29/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:4751s | GPU cuda2:5237/24564MiB:28%;cuda3:2765/24564MiB:95%260814-07:02:57 (UTC+8)
166tasks done=236/961 running=1 failed=0 blocked=0 pending=724 | results done=29/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:4570s | GPU cuda2:5237/24564MiB:27%;cuda3:2765/24564MiB:97%260814-06:59:56 (UTC+8)
165tasks done=236/961 running=1 failed=0 blocked=0 pending=724 | results done=29/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:4389s | GPU cuda2:5237/24564MiB:25%;cuda3:2765/24564MiB:94%260814-06:56:55 (UTC+8)
164tasks done=236/961 running=1 failed=0 blocked=0 pending=724 | results done=29/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:4208s | GPU cuda2:5237/24564MiB:28%;cuda3:2765/24564MiB:97%260814-06:53:54 (UTC+8)
163tasks done=236/961 running=1 failed=0 blocked=0 pending=724 | results done=29/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:4027s | GPU cuda2:5237/24564MiB:78%;cuda3:2765/24564MiB:96%260814-06:50:53 (UTC+8)
162tasks done=236/961 running=1 failed=0 blocked=0 pending=724 | results done=29/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:3847s | GPU cuda2:5237/24564MiB:29%;cuda3:2765/24564MiB:95%260814-06:47:53 (UTC+8)
161tasks done=233/961 running=2 failed=0 blocked=0 pending=726 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:3665s;cuda3:external_downstream:run__DuoKD__llama31_4b__cora__s666__gcn3:2s | GPU cuda2:5237/24564MiB:34%;cuda3:3152/24564MiB:97%260814-06:44:52 (UTC+8)
160tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:3485s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:4843s | GPU cuda2:5343/24564MiB:98%;cuda3:7392/24564MiB:100%260814-06:41:50 (UTC+8)
159tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:3304s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:4662s | GPU cuda2:5343/24564MiB:98%;cuda3:7392/24564MiB:100%260814-06:38:50 (UTC+8)
158tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:3123s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:4482s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-06:35:49 (UTC+8)
157tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:2943s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:4301s | GPU cuda2:5343/24564MiB:99%;cuda3:7392/24564MiB:100%260814-06:32:49 (UTC+8)
156tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:2762s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:4120s | GPU cuda2:5343/24564MiB:99%;cuda3:7392/24564MiB:100%260814-06:29:48 (UTC+8)
155tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:2582s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:3940s | GPU cuda2:5343/24564MiB:98%;cuda3:7392/24564MiB:100%260814-06:26:47 (UTC+8)
154tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:2401s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:3759s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-06:23:47 (UTC+8)
153tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:2220s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:3579s | GPU cuda2:5343/24564MiB:85%;cuda3:7392/24564MiB:100%260814-06:20:46 (UTC+8)
152tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:2039s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:3398s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-06:17:46 (UTC+8)
151tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:1859s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:3217s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-06:14:45 (UTC+8)
150tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:1678s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:3036s | GPU cuda2:5343/24564MiB:96%;cuda3:7392/24564MiB:100%260814-06:11:44 (UTC+8)
149tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:1497s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:2856s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-06:08:43 (UTC+8)
148tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:1317s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:2675s | GPU cuda2:5343/24564MiB:99%;cuda3:7392/24564MiB:100%260814-06:05:43 (UTC+8)
147tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:1136s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:2495s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-06:02:42 (UTC+8)
146tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:956s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:2314s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-05:59:41 (UTC+8)
145tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:775s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:2134s | GPU cuda2:5343/24564MiB:99%;cuda3:7392/24564MiB:100%260814-05:56:41 (UTC+8)
144tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:595s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:1953s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-05:53:40 (UTC+8)
143tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:414s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:1773s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-05:50:40 (UTC+8)
142tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:233s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:1592s | GPU cuda2:5343/24564MiB:97%;cuda3:7392/24564MiB:100%260814-05:47:39 (UTC+8)
141tasks done=231/961 running=2 failed=0 blocked=0 pending=728 | results done=28/120 | RUNNING cuda2:duo_bank:duo_bank__llama31_4b__cora__s888:53s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:1411s | GPU cuda2:5323/24564MiB:97%;cuda3:7392/24564MiB:100%260814-05:44:38 (UTC+8)
140tasks done=221/961 running=2 failed=0 blocked=0 pending=738 | results done=21/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__cora__s888:358s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:1231s | GPU cuda2:3181/24564MiB:14%;cuda3:7392/24564MiB:97%260814-05:41:38 (UTC+8)
139tasks done=221/961 running=2 failed=0 blocked=0 pending=738 | results done=21/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__cora__s888:177s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:1050s | GPU cuda2:6125/24564MiB:96%;cuda3:7392/24564MiB:100%260814-05:38:37 (UTC+8)
138tasks done=220/961 running=2 failed=0 blocked=0 pending=739 | results done=21/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__cora__s888:355s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:869s | GPU cuda2:6125/24564MiB:97%;cuda3:7392/24564MiB:100%260814-05:35:37 (UTC+8)
137tasks done=220/961 running=2 failed=0 blocked=0 pending=739 | results done=21/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__cora__s888:175s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:689s | GPU cuda2:6125/24564MiB:100%;cuda3:7392/24564MiB:100%260814-05:32:36 (UTC+8)
136tasks done=218/961 running=2 failed=0 blocked=0 pending=741 | results done=21/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-512-b__llama31_4b__cora__s888:199s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:508s | GPU cuda2:5565/24564MiB:100%;cuda3:7392/24564MiB:100%260814-05:29:35 (UTC+8)
135tasks done=218/961 running=2 failed=0 blocked=0 pending=741 | results done=21/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-512-b__llama31_4b__cora__s888:19s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:328s | GPU cuda2:5361/24564MiB:97%;cuda3:7392/24564MiB:100%260814-05:26:35 (UTC+8)
134tasks done=217/961 running=2 failed=0 blocked=0 pending=742 | results done=21/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-512-a__llama31_4b__cora__s888:52s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s666:147s | GPU cuda2:5539/24564MiB:98%;cuda3:7392/24564MiB:100%260814-05:23:34 (UTC+8)
133tasks done=212/961 running=2 failed=0 blocked=0 pending=747 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-256-b__llama31_4b__cora__s888:19s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:4986s | GPU cuda2:5259/24564MiB:98%;cuda3:7392/24564MiB:100%260814-05:20:34 (UTC+8)
132tasks done=209/961 running=2 failed=0 blocked=0 pending=750 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-128-b__llama31_4b__cora__s888:109s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:4805s | GPU cuda2:5259/24564MiB:98%;cuda3:7392/24564MiB:100%260814-05:17:33 (UTC+8)
131tasks done=208/961 running=2 failed=0 blocked=0 pending=751 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-128-a__llama31_4b__cora__s888:50s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:4624s | GPU cuda2:5259/24564MiB:99%;cuda3:7392/24564MiB:100%260814-05:14:32 (UTC+8)
130tasks done=206/961 running=2 failed=0 blocked=0 pending=753 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s888:470s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:4444s | GPU cuda2:6139/24564MiB:100%;cuda3:7392/24564MiB:100%260814-05:11:31 (UTC+8)
129tasks done=206/961 running=2 failed=0 blocked=0 pending=753 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s888:290s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:4263s | GPU cuda2:6139/24564MiB:14%;cuda3:7392/24564MiB:100%260814-05:08:30 (UTC+8)
128tasks done=206/961 running=2 failed=0 blocked=0 pending=753 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s888:109s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:4082s | GPU cuda2:6139/24564MiB:15%;cuda3:7392/24564MiB:99%260814-05:05:30 (UTC+8)
127tasks done=205/961 running=2 failed=0 blocked=0 pending=754 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s888:530s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:3901s | GPU cuda2:6139/24564MiB:98%;cuda3:7392/24564MiB:100%260814-05:02:29 (UTC+8)
126tasks done=205/961 running=2 failed=0 blocked=0 pending=754 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s888:350s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:3721s | GPU cuda2:6139/24564MiB:15%;cuda3:7392/24564MiB:100%260814-04:59:28 (UTC+8)
125tasks done=205/961 running=2 failed=0 blocked=0 pending=754 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s888:169s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:3540s | GPU cuda2:6139/24564MiB:97%;cuda3:7392/24564MiB:100%260814-04:56:27 (UTC+8)
124tasks done=203/961 running=2 failed=0 blocked=0 pending=756 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-b__llama31_4b__cora__s888:426s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:3359s | GPU cuda2:5579/24564MiB:97%;cuda3:7392/24564MiB:100%260814-04:53:27 (UTC+8)
123tasks done=203/961 running=2 failed=0 blocked=0 pending=756 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-b__llama31_4b__cora__s888:245s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:3179s | GPU cuda2:5579/24564MiB:100%;cuda3:7392/24564MiB:100%260814-04:50:26 (UTC+8)
122tasks done=203/961 running=2 failed=0 blocked=0 pending=756 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-b__llama31_4b__cora__s888:64s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:2998s | GPU cuda2:5579/24564MiB:14%;cuda3:7392/24564MiB:100%260814-04:47:25 (UTC+8)
121tasks done=202/961 running=2 failed=0 blocked=0 pending=757 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-a__llama31_4b__cora__s888:316s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:2818s | GPU cuda2:5579/24564MiB:15%;cuda3:7392/24564MiB:100%260814-04:44:25 (UTC+8)
120tasks done=202/961 running=2 failed=0 blocked=0 pending=757 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-a__llama31_4b__cora__s888:135s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:2637s | GPU cuda2:5579/24564MiB:15%;cuda3:7392/24564MiB:100%260814-04:41:24 (UTC+8)
119tasks done=200/961 running=2 failed=0 blocked=0 pending=759 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-b__llama31_4b__cora__s888:320s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:2456s | GPU cuda2:5299/24564MiB:15%;cuda3:7392/24564MiB:100%260814-04:38:23 (UTC+8)
118tasks done=200/961 running=2 failed=0 blocked=0 pending=759 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-b__llama31_4b__cora__s888:139s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:2275s | GPU cuda2:5299/24564MiB:97%;cuda3:7392/24564MiB:100%260814-04:35:22 (UTC+8)
117tasks done=199/961 running=2 failed=0 blocked=0 pending=760 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-a__llama31_4b__cora__s888:324s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:2094s | GPU cuda2:5299/24564MiB:98%;cuda3:7392/24564MiB:100%260814-04:32:22 (UTC+8)
116tasks done=199/961 running=2 failed=0 blocked=0 pending=760 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-a__llama31_4b__cora__s888:144s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:1914s | GPU cuda2:5299/24564MiB:100%;cuda3:7392/24564MiB:100%260814-04:29:21 (UTC+8)
115tasks done=197/961 running=2 failed=0 blocked=0 pending=762 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-b__llama31_4b__cora__s888:300s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:1733s | GPU cuda2:5259/24564MiB:99%;cuda3:7392/24564MiB:100%260814-04:26:21 (UTC+8)
114tasks done=197/961 running=2 failed=0 blocked=0 pending=762 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-b__llama31_4b__cora__s888:119s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:1552s | GPU cuda2:5259/24564MiB:14%;cuda3:7392/24564MiB:100%260814-04:23:20 (UTC+8)
113tasks done=196/961 running=2 failed=0 blocked=0 pending=763 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-a__llama31_4b__cora__s888:262s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:1371s | GPU cuda2:5151/24564MiB:97%;cuda3:7392/24564MiB:100%260814-04:20:19 (UTC+8)
112tasks done=196/961 running=2 failed=0 blocked=0 pending=763 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-a__llama31_4b__cora__s888:81s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:1191s | GPU cuda2:5235/24564MiB:14%;cuda3:7392/24564MiB:100%260814-04:17:18 (UTC+8)
111tasks done=194/961 running=2 failed=0 blocked=0 pending=765 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__cora__s888:313s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:1010s | GPU cuda2:6115/24564MiB:100%;cuda3:7392/24564MiB:100%260814-04:14:17 (UTC+8)
110tasks done=194/961 running=2 failed=0 blocked=0 pending=765 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__cora__s888:132s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:829s | GPU cuda2:6115/24564MiB:100%;cuda3:7392/24564MiB:100%260814-04:11:16 (UTC+8)
109tasks done=193/961 running=2 failed=0 blocked=0 pending=766 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-a__llama31_4b__cora__s888:360s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:648s | GPU cuda2:6115/24564MiB:100%;cuda3:7392/24564MiB:100%260814-04:08:16 (UTC+8)
108tasks done=193/961 running=2 failed=0 blocked=0 pending=766 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-a__llama31_4b__cora__s888:180s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:468s | GPU cuda2:6115/24564MiB:96%;cuda3:7392/24564MiB:100%260814-04:05:15 (UTC+8)
107tasks done=192/961 running=2 failed=0 blocked=0 pending=767 | results done=20/120 | RUNNING cuda2:pair_audit:pair_audit__Semantic-512__llama31_4b__cora__s888:1s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:287s | GPU cuda2:683/24564MiB:15%;cuda3:7392/24564MiB:100%260814-04:02:14 (UTC+8)
106tasks done=191/961 running=2 failed=0 blocked=0 pending=768 | results done=20/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-b__llama31_4b__cora__s888:64s;cuda3:duo_bank:duo_bank__llama31_4b__cora__s42:106s | GPU cuda2:5555/24564MiB:98%;cuda3:7392/24564MiB:100%260814-03:59:14 (UTC+8)
105tasks done=187/961 running=2 failed=0 blocked=0 pending=772 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-a__llama31_4b__cora__s888:128s;cuda3:lingu_bank:lingu_bank__llama31_4b__cora__s888:443s | GPU cuda2:5553/24564MiB:100%;cuda3:8256/24564MiB:100%260814-03:56:13 (UTC+8)
104tasks done=185/961 running=2 failed=0 blocked=0 pending=774 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-128-b__llama31_4b__cora__s888:86s;cuda3:lingu_bank:lingu_bank__llama31_4b__cora__s888:262s | GPU cuda2:5233/24564MiB:98%;cuda3:8236/24564MiB:97%260814-03:53:12 (UTC+8)
103tasks done=184/961 running=2 failed=0 blocked=0 pending=775 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-128-a__llama31_4b__cora__s888:43s;cuda3:lingu_bank:lingu_bank__llama31_4b__cora__s888:81s | GPU cuda2:5233/24564MiB:97%;cuda3:8236/24564MiB:100%260814-03:50:11 (UTC+8)
102tasks done=181/961 running=2 failed=0 blocked=0 pending=778 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-b__llama31_4b__cora__s888:269s;cuda3:sft:sft__lingu__llama31_4b__cora__s888:1263s | GPU cuda2:6111/24564MiB:100%;cuda3:11220/24564MiB:100%260814-03:47:10 (UTC+8)
101tasks done=181/961 running=2 failed=0 blocked=0 pending=778 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-b__llama31_4b__cora__s888:89s;cuda3:sft:sft__lingu__llama31_4b__cora__s888:1082s | GPU cuda2:6111/24564MiB:96%;cuda3:11220/24564MiB:100%260814-03:44:10 (UTC+8)
100tasks done=180/961 running=2 failed=0 blocked=0 pending=779 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-a__llama31_4b__cora__s888:315s;cuda3:sft:sft__lingu__llama31_4b__cora__s888:901s | GPU cuda2:6319/24564MiB:96%;cuda3:11220/24564MiB:100%260814-03:41:09 (UTC+8)
99tasks done=180/961 running=2 failed=0 blocked=0 pending=779 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-a__llama31_4b__cora__s888:134s;cuda3:sft:sft__lingu__llama31_4b__cora__s888:720s | GPU cuda2:6319/24564MiB:100%;cuda3:11220/24564MiB:100%260814-03:38:08 (UTC+8)
98tasks done=178/961 running=2 failed=0 blocked=0 pending=781 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-b__llama31_4b__cora__s888:197s;cuda3:sft:sft__lingu__llama31_4b__cora__s888:539s | GPU cuda2:5759/24564MiB:100%;cuda3:11220/24564MiB:100%260814-03:35:07 (UTC+8)
97tasks done=178/961 running=2 failed=0 blocked=0 pending=781 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-b__llama31_4b__cora__s888:16s;cuda3:sft:sft__lingu__llama31_4b__cora__s888:359s | GPU cuda2:5423/24564MiB:16%;cuda3:11220/24564MiB:100%260814-03:32:06 (UTC+8)
96tasks done=177/961 running=2 failed=0 blocked=0 pending=782 | results done=19/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-a__llama31_4b__cora__s888:76s;cuda3:sft:sft__lingu__llama31_4b__cora__s888:178s | GPU cuda2:5759/24564MiB:100%;cuda3:11218/24564MiB:100%260814-03:29:06 (UTC+8)
95tasks done=174/961 running=2 failed=0 blocked=0 pending=785 | results done=18/120 | RUNNING cuda2:trace_context:trace_bank__Random-256-b__llama31_4b__cora__s888:66s;cuda3:rci_downstream:run__RCI_Unconditioned__llama31_4b__cora__s666__gcn3:22s | GPU cuda2:5479/24564MiB:96%;cuda3:3362/24564MiB:99%260814-03:26:05 (UTC+8)
94tasks done=165/961 running=2 failed=0 blocked=0 pending=794 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Random-256-a__llama31_4b__cora__s888:55s;cuda3:pair_audit:pair_audit__HopQuota-1024__llama31_4b__cora__s666:2s | GPU cuda2:5479/24564MiB:98%;cuda3:3152/24564MiB:97%260814-03:23:04 (UTC+8)
93tasks done=162/961 running=2 failed=0 blocked=0 pending=797 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Random-128-b__llama31_4b__cora__s888:10s;cuda3:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__cora__s666:293s | GPU cuda2:5301/24564MiB:29%;cuda3:8176/24564MiB:100%260814-03:20:03 (UTC+8)
92tasks done=159/961 running=2 failed=0 blocked=0 pending=800 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__cora__s888:371s;cuda3:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__cora__s666:112s | GPU cuda2:6319/24564MiB:96%;cuda3:8156/24564MiB:100%260814-03:17:03 (UTC+8)
91tasks done=158/961 running=2 failed=0 blocked=0 pending=801 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__cora__s888:190s;cuda3:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__cora__s666:404s | GPU cuda2:6319/24564MiB:100%;cuda3:8176/24564MiB:100%260814-03:14:02 (UTC+8)
90tasks done=158/961 running=2 failed=0 blocked=0 pending=801 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__cora__s888:10s;cuda3:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__cora__s666:224s | GPU cuda2:5301/24564MiB:29%;cuda3:8176/24564MiB:100%260814-03:11:02 (UTC+8)
89tasks done=157/961 running=2 failed=0 blocked=0 pending=802 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-a__llama31_4b__cora__s888:236s;cuda3:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__cora__s666:43s | GPU cuda2:6319/24564MiB:100%;cuda3:8136/24564MiB:100%260814-03:08:01 (UTC+8)
88tasks done=155/961 running=2 failed=0 blocked=0 pending=804 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-a__llama31_4b__cora__s888:55s;cuda3:trace_context:trace_bank__HopQuota-512-b__llama31_4b__cora__s666:145s | GPU cuda2:6319/24564MiB:100%;cuda3:7616/24564MiB:100%260814-03:05:00 (UTC+8)
87tasks done=152/961 running=2 failed=0 blocked=0 pending=807 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-256-b__llama31_4b__cora__s888:45s;cuda3:trace_context:trace_bank__HopQuota-512-a__llama31_4b__cora__s666:245s | GPU cuda2:5479/24564MiB:41%;cuda3:7616/24564MiB:100%260814-03:01:59 (UTC+8)
86tasks done=151/961 running=2 failed=0 blocked=0 pending=808 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-256-a__llama31_4b__cora__s888:33s;cuda3:trace_context:trace_bank__HopQuota-512-a__llama31_4b__cora__s666:64s | GPU cuda2:5479/24564MiB:98%;cuda3:7596/24564MiB:100%260814-02:58:59 (UTC+8)
85tasks done=146/961 running=2 failed=0 blocked=0 pending=813 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-128-a__llama31_4b__cora__s888:120s;cuda3:trace_context:trace_bank__HopQuota-256-b__llama31_4b__cora__s666:75s | GPU cuda2:5439/24564MiB:99%;cuda3:7336/24564MiB:100%260814-02:55:58 (UTC+8)
84tasks done=143/961 running=2 failed=0 blocked=0 pending=816 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-256-b__llama31_4b__cora__s888:114s;cuda3:trace_context:trace_bank__HopQuota-256-a__llama31_4b__cora__s666:83s | GPU cuda2:5479/24564MiB:98%;cuda3:7336/24564MiB:100%260814-02:52:57 (UTC+8)
83tasks done=140/961 running=2 failed=0 blocked=0 pending=819 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-256-a__llama31_4b__cora__s888:106s;cuda3:trace_context:trace_bank__HopQuota-128-b__llama31_4b__cora__s666:62s | GPU cuda2:5479/24564MiB:98%;cuda3:7316/24564MiB:100%260814-02:49:57 (UTC+8)
82tasks done=137/961 running=2 failed=0 blocked=0 pending=822 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-b__llama31_4b__cora__s888:167s;cuda3:trace_context:trace_bank__HopQuota-128-a__llama31_4b__cora__s666:39s | GPU cuda2:5759/24564MiB:97%;cuda3:7316/24564MiB:100%260814-02:46:56 (UTC+8)
81tasks done=134/961 running=2 failed=0 blocked=0 pending=825 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-a__llama31_4b__cora__s888:227s;cuda3:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s666:589s | GPU cuda2:5759/24564MiB:100%;cuda3:8196/24564MiB:100%260814-02:43:55 (UTC+8)
80tasks done=134/961 running=2 failed=0 blocked=0 pending=825 | results done=12/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-a__llama31_4b__cora__s888:46s;cuda3:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s666:408s | GPU cuda2:5759/24564MiB:98%;cuda3:8196/24564MiB:100%260814-02:40:55 (UTC+8)
79tasks done=132/961 running=2 failed=0 blocked=0 pending=827 | results done=12/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__cora__s888:638s;cuda3:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s666:227s | GPU cuda2:8821/24564MiB:100%;cuda3:8196/24564MiB:100%260814-02:37:54 (UTC+8)
78tasks done=132/961 running=2 failed=0 blocked=0 pending=827 | results done=12/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__cora__s888:458s;cuda3:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s666:47s | GPU cuda2:8821/24564MiB:100%;cuda3:8196/24564MiB:100%260814-02:34:53 (UTC+8)
77tasks done=131/961 running=2 failed=0 blocked=0 pending=828 | results done=12/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__cora__s888:277s;cuda3:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s666:599s | GPU cuda2:8821/24564MiB:99%;cuda3:8196/24564MiB:96%260814-02:31:53 (UTC+8)
76tasks done=131/961 running=2 failed=0 blocked=0 pending=828 | results done=12/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__cora__s888:96s;cuda3:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s666:418s | GPU cuda2:8821/24564MiB:99%;cuda3:8196/24564MiB:100%260814-02:28:52 (UTC+8)
75tasks done=128/961 running=2 failed=0 blocked=0 pending=831 | results done=11/120 | RUNNING cuda2:lingu_bank:lingu_bank__llama31_4b__cora__s666:314s;cuda3:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s666:237s | GPU cuda2:6379/24564MiB:100%;cuda3:8196/24564MiB:100%260814-02:25:51 (UTC+8)
74tasks done=128/961 running=2 failed=0 blocked=0 pending=831 | results done=11/120 | RUNNING cuda2:lingu_bank:lingu_bank__llama31_4b__cora__s666:133s;cuda3:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s666:56s | GPU cuda2:6359/24564MiB:15%;cuda3:8196/24564MiB:94%260814-02:22:50 (UTC+8)
73tasks done=125/961 running=2 failed=0 blocked=0 pending=834 | results done=11/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__cora__s666:982s;cuda3:trace_context:trace_bank__Diverse-512-b__llama31_4b__cora__s666:384s | GPU cuda2:9923/24564MiB:100%;cuda3:7636/24564MiB:100%260814-02:19:50 (UTC+8)
72tasks done=125/961 running=2 failed=0 blocked=0 pending=834 | results done=11/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__cora__s666:801s;cuda3:trace_context:trace_bank__Diverse-512-b__llama31_4b__cora__s666:204s | GPU cuda2:9923/24564MiB:100%;cuda3:7636/24564MiB:97%260814-02:16:49 (UTC+8)
71tasks done=125/961 running=2 failed=0 blocked=0 pending=834 | results done=11/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__cora__s666:621s;cuda3:trace_context:trace_bank__Diverse-512-b__llama31_4b__cora__s666:23s | GPU cuda2:9323/24564MiB:100%;cuda3:7300/24564MiB:97%260814-02:13:49 (UTC+8)
70tasks done=124/961 running=2 failed=0 blocked=0 pending=835 | results done=11/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__cora__s666:440s;cuda3:trace_context:trace_bank__Diverse-512-a__llama31_4b__cora__s666:352s | GPU cuda2:9323/24564MiB:100%;cuda3:7636/24564MiB:98%260814-02:10:48 (UTC+8)
69tasks done=124/961 running=2 failed=0 blocked=0 pending=835 | results done=11/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__cora__s666:260s;cuda3:trace_context:trace_bank__Diverse-512-a__llama31_4b__cora__s666:171s | GPU cuda2:9323/24564MiB:97%;cuda3:7636/24564MiB:100%260814-02:07:47 (UTC+8)
68tasks done=122/961 running=2 failed=0 blocked=0 pending=837 | results done=11/120 | RUNNING cuda2:sft:sft__lingu__llama31_4b__cora__s666:79s;cuda3:trace_context:trace_bank__Diverse-256-b__llama31_4b__cora__s666:420s | GPU cuda2:9303/24564MiB:100%;cuda3:7356/24564MiB:100%260814-02:04:47 (UTC+8)
67tasks done=114/961 running=2 failed=0 blocked=0 pending=845 | results done=4/120 | RUNNING cuda2:trace_analysis:trace_analysis__llama31_4b__cora__s42:27s;cuda3:trace_context:trace_bank__Diverse-256-b__llama31_4b__cora__s666:239s | GPU cuda2:1485/24564MiB:15%;cuda3:7356/24564MiB:99%260814-02:01:46 (UTC+8)
66tasks done=112/961 running=2 failed=0 blocked=0 pending=847 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__cora__s42:208s;cuda3:trace_context:trace_bank__Diverse-256-b__llama31_4b__cora__s666:58s | GPU cuda2:6299/24564MiB:100%;cuda3:7356/24564MiB:98%260814-01:58:45 (UTC+8)
65tasks done=111/961 running=2 failed=0 blocked=0 pending=848 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-b__llama31_4b__cora__s42:27s;cuda3:trace_context:trace_bank__Diverse-256-a__llama31_4b__cora__s666:290s | GPU cuda2:6259/24564MiB:100%;cuda3:7356/24564MiB:100%260814-01:55:44 (UTC+8)
64tasks done=110/961 running=2 failed=0 blocked=0 pending=849 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__cora__s42:207s;cuda3:trace_context:trace_bank__Diverse-256-a__llama31_4b__cora__s666:110s | GPU cuda2:6299/24564MiB:100%;cuda3:7356/24564MiB:97%260814-01:52:44 (UTC+8)
63tasks done=108/961 running=2 failed=0 blocked=0 pending=851 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-1024-a__llama31_4b__cora__s42:27s;cuda3:trace_context:trace_bank__Diverse-128-b__llama31_4b__cora__s666:301s | GPU cuda2:6259/24564MiB:97%;cuda3:7316/24564MiB:98%260814-01:49:43 (UTC+8)
62tasks done=106/961 running=2 failed=0 blocked=0 pending=853 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-512-b__llama31_4b__cora__s42:63s;cuda3:trace_context:trace_bank__Diverse-128-b__llama31_4b__cora__s666:120s | GPU cuda2:5719/24564MiB:98%;cuda3:7316/24564MiB:96%260814-01:46:43 (UTC+8)
61tasks done=104/961 running=2 failed=0 blocked=0 pending=855 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-512-a__llama31_4b__cora__s42:97s;cuda3:trace_context:trace_bank__Diverse-128-a__llama31_4b__cora__s666:309s | GPU cuda2:5739/24564MiB:98%;cuda3:7316/24564MiB:98%260814-01:43:42 (UTC+8)
60tasks done=102/961 running=2 failed=0 blocked=0 pending=857 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-256-b__llama31_4b__cora__s42:63s;cuda3:trace_context:trace_bank__Diverse-128-a__llama31_4b__cora__s666:128s | GPU cuda2:5459/24564MiB:98%;cuda3:7316/24564MiB:100%260814-01:40:41 (UTC+8)
59tasks done=99/961 running=2 failed=0 blocked=0 pending=860 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-256-a__llama31_4b__cora__s42:29s;cuda3:trace_context:trace_bank__Semantic-1024-b__llama31_4b__cora__s666:486s | GPU cuda2:5439/24564MiB:99%;cuda3:8196/24564MiB:100%260814-01:37:41 (UTC+8)
58tasks done=96/961 running=2 failed=0 blocked=0 pending=863 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__HopQuota-128-a__llama31_4b__cora__s42:95s;cuda3:trace_context:trace_bank__Semantic-1024-b__llama31_4b__cora__s666:305s | GPU cuda2:5439/24564MiB:98%;cuda3:8196/24564MiB:100%260814-01:34:40 (UTC+8)
57tasks done=94/961 running=2 failed=0 blocked=0 pending=865 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s42:518s;cuda3:trace_context:trace_bank__Semantic-1024-b__llama31_4b__cora__s666:125s | GPU cuda2:6319/24564MiB:16%;cuda3:8196/24564MiB:100%260814-01:31:39 (UTC+8)
56tasks done=93/961 running=2 failed=0 blocked=0 pending=866 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s42:337s;cuda3:trace_context:trace_bank__Semantic-1024-a__llama31_4b__cora__s666:480s | GPU cuda2:6319/24564MiB:95%;cuda3:8196/24564MiB:100%260814-01:28:38 (UTC+8)
55tasks done=93/961 running=2 failed=0 blocked=0 pending=866 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-b__llama31_4b__cora__s42:156s;cuda3:trace_context:trace_bank__Semantic-1024-a__llama31_4b__cora__s666:300s | GPU cuda2:6319/24564MiB:16%;cuda3:8196/24564MiB:100%260814-01:25:38 (UTC+8)
54tasks done=92/961 running=2 failed=0 blocked=0 pending=867 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s42:576s;cuda3:trace_context:trace_bank__Semantic-1024-a__llama31_4b__cora__s666:119s | GPU cuda2:6319/24564MiB:100%;cuda3:8196/24564MiB:100%260814-01:22:37 (UTC+8)
53tasks done=90/961 running=2 failed=0 blocked=0 pending=869 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s42:396s;cuda3:trace_context:trace_bank__Semantic-512-b__llama31_4b__cora__s666:257s | GPU cuda2:6319/24564MiB:100%;cuda3:7636/24564MiB:100%260814-01:19:37 (UTC+8)
52tasks done=90/961 running=2 failed=0 blocked=0 pending=869 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s42:215s;cuda3:trace_context:trace_bank__Semantic-512-b__llama31_4b__cora__s666:76s | GPU cuda2:6319/24564MiB:15%;cuda3:7636/24564MiB:100%260814-01:16:36 (UTC+8)
51tasks done=89/961 running=2 failed=0 blocked=0 pending=870 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-1024-a__llama31_4b__cora__s42:34s;cuda3:trace_context:trace_bank__Semantic-512-a__llama31_4b__cora__s666:213s | GPU cuda2:6319/24564MiB:97%;cuda3:7636/24564MiB:100%260814-01:13:35 (UTC+8)
50tasks done=87/961 running=2 failed=0 blocked=0 pending=872 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-b__llama31_4b__cora__s42:296s;cuda3:trace_context:trace_bank__Semantic-512-a__llama31_4b__cora__s666:32s | GPU cuda2:5759/24564MiB:97%;cuda3:7636/24564MiB:100%260814-01:10:34 (UTC+8)
49tasks done=85/961 running=2 failed=0 blocked=0 pending=874 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-b__llama31_4b__cora__s42:115s;cuda3:trace_context:trace_bank__Semantic-128-b__llama31_4b__cora__s666:29s | GPU cuda2:5759/24564MiB:16%;cuda3:7316/24564MiB:100%260814-01:07:34 (UTC+8)
48tasks done=83/961 running=2 failed=0 blocked=0 pending=876 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-a__llama31_4b__cora__s42:368s;cuda3:trace_context:trace_bank__Semantic-128-a__llama31_4b__cora__s666:24s | GPU cuda2:5759/24564MiB:100%;cuda3:7316/24564MiB:98%260814-01:04:33 (UTC+8)
47tasks done=81/961 running=2 failed=0 blocked=0 pending=878 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-a__llama31_4b__cora__s42:187s;cuda3:trace_context:trace_bank__Random-1024-b__llama31_4b__cora__s666:376s | GPU cuda2:5759/24564MiB:98%;cuda3:8196/24564MiB:100%260814-01:01:33 (UTC+8)
46tasks done=81/961 running=2 failed=0 blocked=0 pending=878 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-512-a__llama31_4b__cora__s42:6s;cuda3:trace_context:trace_bank__Random-1024-b__llama31_4b__cora__s666:195s | GPU cuda2:1275/24564MiB:16%;cuda3:8196/24564MiB:100%260814-00:58:32 (UTC+8)
45tasks done=79/961 running=2 failed=0 blocked=0 pending=880 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-b__llama31_4b__cora__s42:192s;cuda3:trace_context:trace_bank__Random-1024-b__llama31_4b__cora__s666:14s | GPU cuda2:5479/24564MiB:98%;cuda3:7214/24564MiB:98%260814-00:55:31 (UTC+8)
44tasks done=78/961 running=2 failed=0 blocked=0 pending=881 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-b__llama31_4b__cora__s42:12s;cuda3:trace_context:trace_bank__Random-1024-a__llama31_4b__cora__s666:365s | GPU cuda2:5313/24564MiB:30%;cuda3:8196/24564MiB:100%260814-00:52:30 (UTC+8)
43tasks done=77/961 running=2 failed=0 blocked=0 pending=882 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-a__llama31_4b__cora__s42:193s;cuda3:trace_context:trace_bank__Random-1024-a__llama31_4b__cora__s666:184s | GPU cuda2:5479/24564MiB:98%;cuda3:8196/24564MiB:100%260814-00:49:30 (UTC+8)
42tasks done=77/961 running=2 failed=0 blocked=0 pending=882 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-256-a__llama31_4b__cora__s42:12s;cuda3:trace_context:trace_bank__Random-1024-a__llama31_4b__cora__s666:3s | GPU cuda2:5315/24564MiB:28%;cuda3:3152/24564MiB:97%260814-00:46:29 (UTC+8)
41tasks done=73/961 running=2 failed=0 blocked=0 pending=886 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-b__llama31_4b__cora__s42:164s;cuda3:trace_context:trace_bank__Random-512-b__llama31_4b__cora__s666:135s | GPU cuda2:5439/24564MiB:15%;cuda3:7636/24564MiB:100%260814-00:43:28 (UTC+8)
40tasks done=71/961 running=2 failed=0 blocked=0 pending=888 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-a__llama31_4b__cora__s42:312s;cuda3:trace_context:trace_bank__Random-512-a__llama31_4b__cora__s666:267s | GPU cuda2:5439/24564MiB:16%;cuda3:7636/24564MiB:100%260814-00:40:26 (UTC+8)
39tasks done=71/961 running=2 failed=0 blocked=0 pending=888 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Diverse-128-a__llama31_4b__cora__s42:131s;cuda3:trace_context:trace_bank__Random-512-a__llama31_4b__cora__s666:86s | GPU cuda2:5439/24564MiB:48%;cuda3:7636/24564MiB:100%260814-00:37:26 (UTC+8)
38tasks done=67/961 running=2 failed=0 blocked=0 pending=892 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__cora__s42:363s;cuda3:trace_context:trace_bank__Random-256-b__llama31_4b__cora__s666:127s | GPU cuda2:6319/24564MiB:100%;cuda3:7356/24564MiB:100%260814-00:34:25 (UTC+8)
37tasks done=66/961 running=2 failed=0 blocked=0 pending=893 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__cora__s42:183s;cuda3:trace_context:trace_bank__Random-256-a__llama31_4b__cora__s666:165s | GPU cuda2:6319/24564MiB:100%;cuda3:7356/24564MiB:100%260814-00:31:24 (UTC+8)
36tasks done=64/961 running=2 failed=0 blocked=0 pending=895 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-b__llama31_4b__cora__s42:2s;cuda3:trace_context:trace_bank__Random-128-b__llama31_4b__cora__s666:157s | GPU cuda2:1275/24564MiB:15%;cuda3:7316/24564MiB:100%260814-00:28:23 (UTC+8)
35tasks done=62/961 running=2 failed=0 blocked=0 pending=897 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-a__llama31_4b__cora__s42:232s;cuda3:trace_context:trace_bank__Random-128-a__llama31_4b__cora__s666:148s | GPU cuda2:6319/24564MiB:100%;cuda3:7316/24564MiB:100%260814-00:25:22 (UTC+8)
34tasks done=60/961 running=2 failed=0 blocked=0 pending=899 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-1024-a__llama31_4b__cora__s42:50s;cuda3:trace_context:trace_bank__Full-1024-b__llama31_4b__cora__s666:498s | GPU cuda2:6319/24564MiB:96%;cuda3:8196/24564MiB:100%260814-00:22:22 (UTC+8)
33tasks done=58/961 running=2 failed=0 blocked=0 pending=901 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-b__llama31_4b__cora__s42:116s;cuda3:trace_context:trace_bank__Full-1024-b__llama31_4b__cora__s666:318s | GPU cuda2:5759/24564MiB:98%;cuda3:8196/24564MiB:100%260814-00:19:21 (UTC+8)
32tasks done=57/961 running=2 failed=0 blocked=0 pending=902 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-512-a__llama31_4b__cora__s42:180s;cuda3:trace_context:trace_bank__Full-1024-b__llama31_4b__cora__s666:137s | GPU cuda2:5759/24564MiB:100%;cuda3:8196/24564MiB:100%260814-00:16:20 (UTC+8)
31tasks done=55/961 running=2 failed=0 blocked=0 pending=904 | results done=4/120 | RUNNING cuda2:pair_audit:pair_audit__Semantic-128__llama31_4b__cora__s42:1s;cuda3:trace_context:trace_bank__Full-1024-a__llama31_4b__cora__s666:487s | GPU cuda2:902/24564MiB:15%;cuda3:8196/24564MiB:100%260814-00:13:20 (UTC+8)
30tasks done=53/961 running=2 failed=0 blocked=0 pending=906 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-128-a__llama31_4b__cora__s42:95s;cuda3:trace_context:trace_bank__Full-1024-a__llama31_4b__cora__s666:306s | GPU cuda2:5439/24564MiB:36%;cuda3:8196/24564MiB:100%260814-00:10:19 (UTC+8)
29tasks done=51/961 running=2 failed=0 blocked=0 pending=908 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-b__llama31_4b__cora__s42:323s;cuda3:trace_context:trace_bank__Full-1024-a__llama31_4b__cora__s666:125s | GPU cuda2:6319/24564MiB:100%;cuda3:8196/24564MiB:100%260814-00:07:18 (UTC+8)
28tasks done=49/961 running=2 failed=0 blocked=0 pending=910 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-b__llama31_4b__cora__s42:142s;cuda3:trace_context:trace_bank__Full-256-b__llama31_4b__cora__s666:165s | GPU cuda2:6319/24564MiB:96%;cuda3:7356/24564MiB:100%260814-00:04:18 (UTC+8)
27tasks done=47/961 running=2 failed=0 blocked=0 pending=912 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-a__llama31_4b__cora__s42:368s;cuda3:trace_context:trace_bank__Full-256-a__llama31_4b__cora__s666:203s | GPU cuda2:6319/24564MiB:96%;cuda3:7356/24564MiB:100%260814-00:01:17 (UTC+8)
26tasks done=47/961 running=2 failed=0 blocked=0 pending=912 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-a__llama31_4b__cora__s42:187s;cuda3:trace_context:trace_bank__Full-256-a__llama31_4b__cora__s666:22s | GPU cuda2:6319/24564MiB:97%;cuda3:7356/24564MiB:100%260813-23:58:16 (UTC+8)
25tasks done=45/961 running=2 failed=0 blocked=0 pending=914 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-1024-a__llama31_4b__cora__s42:6s;cuda3:trace_context:trace_bank__Full-128-b__llama31_4b__cora__s666:14s | GPU cuda2:1275/24564MiB:15%;cuda3:7210/24564MiB:97%260813-23:55:15 (UTC+8)
24tasks done=42/961 running=2 failed=0 blocked=0 pending=917 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-b__llama31_4b__cora__s42:68s;cuda3:trace_context:trace_bank__Full-128-a__llama31_4b__cora__s666:5s | GPU cuda2:5759/24564MiB:98%;cuda3:3152/24564MiB:95%260813-23:52:14 (UTC+8)
23tasks done=39/961 running=2 failed=0 blocked=0 pending=920 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-512-a__llama31_4b__cora__s42:128s;cuda3:trace_context:trace_bank__Semantic-256-b__llama31_4b__cora__s666:50s | GPU cuda2:5759/24564MiB:97%;cuda3:7356/24564MiB:100%260813-23:49:14 (UTC+8)
22tasks done=36/961 running=2 failed=0 blocked=0 pending=923 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-256-b__llama31_4b__cora__s42:118s;cuda3:trace_context:trace_bank__Semantic-256-a__llama31_4b__cora__s666:92s | GPU cuda2:5479/24564MiB:98%;cuda3:7356/24564MiB:100%260813-23:46:13 (UTC+8)
21tasks done=33/961 running=2 failed=0 blocked=0 pending=926 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-256-a__llama31_4b__cora__s42:106s;cuda3:trace_context:trace_bank__Full-512-b__llama31_4b__cora__s666:225s | GPU cuda2:5479/24564MiB:98%;cuda3:7636/24564MiB:100%260813-23:43:12 (UTC+8)
20tasks done=31/961 running=2 failed=0 blocked=0 pending=928 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-128-b__llama31_4b__cora__s42:60s;cuda3:trace_context:trace_bank__Full-512-b__llama31_4b__cora__s666:45s | GPU cuda2:5439/24564MiB:48%;cuda3:7636/24564MiB:99%260813-23:40:11 (UTC+8)
19tasks done=29/961 running=2 failed=0 blocked=0 pending=930 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Random-128-a__llama31_4b__cora__s42:12s;cuda3:trace_context:trace_bank__Full-512-a__llama31_4b__cora__s666:175s | GPU cuda2:5319/24564MiB:30%;cuda3:7636/24564MiB:100%260813-23:37:11 (UTC+8)
18tasks done=26/961 running=2 failed=0 blocked=0 pending=933 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__cora__s42:239s;cuda3:trace_context:trace_bank__Self__llama31_4b__cora__s666:133s | GPU cuda2:6319/24564MiB:100%;cuda3:7316/24564MiB:100%260813-23:34:10 (UTC+8)
17tasks done=25/961 running=2 failed=0 blocked=0 pending=934 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-b__llama31_4b__cora__s42:58s;cuda3:sft:sft__trace__llama31_4b__cora__s666:815s | GPU cuda2:6319/24564MiB:100%;cuda3:10698/24564MiB:100%260813-23:31:09 (UTC+8)
16tasks done=24/961 running=2 failed=0 blocked=0 pending=935 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-a__llama31_4b__cora__s42:283s;cuda3:sft:sft__trace__llama31_4b__cora__s666:634s | GPU cuda2:6319/24564MiB:100%;cuda3:10698/24564MiB:100%260813-23:28:08 (UTC+8)
15tasks done=24/961 running=2 failed=0 blocked=0 pending=935 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Full-1024-a__llama31_4b__cora__s42:103s;cuda3:sft:sft__trace__llama31_4b__cora__s666:453s | GPU cuda2:6319/24564MiB:100%;cuda3:10698/24564MiB:100%260813-23:25:08 (UTC+8)
14tasks done=22/961 running=2 failed=0 blocked=0 pending=937 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Full-256-b__llama31_4b__cora__s42:93s;cuda3:sft:sft__trace__llama31_4b__cora__s666:272s | GPU cuda2:5479/24564MiB:98%;cuda3:10698/24564MiB:100%260813-23:22:07 (UTC+8)
13tasks done=21/961 running=2 failed=0 blocked=0 pending=938 | results done=4/120 | RUNNING cuda2:trace_context:trace_bank__Full-256-a__llama31_4b__cora__s42:80s;cuda3:sft:sft__trace__llama31_4b__cora__s666:90s | GPU cuda2:5479/24564MiB:98%;cuda3:10698/24564MiB:100%260813-23:19:06 (UTC+8)
12tasks done=16/961 running=2 failed=0 blocked=0 pending=943 | results done=3/120 | RUNNING cuda2:trace_context:trace_bank__Full-128-b__llama31_4b__cora__s42:34s;cuda3:lingu_bank:lingu_bank__llama31_4b__cora__s42:421s | GPU cuda2:5439/24564MiB:99%;cuda3:8256/24564MiB:100%260813-23:16:04 (UTC+8)
11tasks done=13/961 running=2 failed=0 blocked=0 pending=946 | results done=3/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-256-b__llama31_4b__cora__s42:162s;cuda3:lingu_bank:lingu_bank__llama31_4b__cora__s42:240s | GPU cuda2:5479/24564MiB:100%;cuda3:8236/24564MiB:100%260813-23:13:04 (UTC+8)
10tasks done=12/961 running=2 failed=0 blocked=0 pending=947 | results done=3/120 | RUNNING cuda2:trace_context:trace_bank__Semantic-256-a__llama31_4b__cora__s42:154s;cuda3:lingu_bank:lingu_bank__llama31_4b__cora__s42:60s | GPU cuda2:5479/24564MiB:98%;cuda3:8236/24564MiB:100%260813-23:10:03 (UTC+8)
9tasks done=9/961 running=2 failed=0 blocked=0 pending=950 | results done=3/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-b__llama31_4b__cora__s42:216s;cuda3:sft:sft__lingu__llama31_4b__cora__s42:1227s | GPU cuda2:5759/24564MiB:99%;cuda3:11220/24564MiB:100%260813-23:07:02 (UTC+8)
8tasks done=9/961 running=2 failed=0 blocked=0 pending=950 | results done=3/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-b__llama31_4b__cora__s42:35s;cuda3:sft:sft__lingu__llama31_4b__cora__s42:1046s | GPU cuda2:5759/24564MiB:100%;cuda3:11220/24564MiB:100%260813-23:04:00 (UTC+8)
7tasks done=8/961 running=2 failed=0 blocked=0 pending=951 | results done=3/120 | RUNNING cuda2:trace_context:trace_bank__Full-512-a__llama31_4b__cora__s42:94s;cuda3:sft:sft__lingu__llama31_4b__cora__s42:864s | GPU cuda2:5759/24564MiB:98%;cuda3:11200/24564MiB:100%260813-23:01:00 (UTC+8)
6tasks done=7/961 running=2 failed=0 blocked=0 pending=952 | results done=3/120 | RUNNING cuda2:trace_context:trace_bank__Self__llama31_4b__cora__s42:20s;cuda3:sft:sft__lingu__llama31_4b__cora__s42:684s | GPU cuda2:5439/24564MiB:98%;cuda3:11200/24564MiB:100%260813-22:57:58 (UTC+8)
5tasks done=6/961 running=2 failed=0 blocked=0 pending=953 | results done=3/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__cora__s42:509s;cuda3:sft:sft__lingu__llama31_4b__cora__s42:503s | GPU cuda2:8821/24564MiB:100%;cuda3:11200/24564MiB:100%260813-22:54:58 (UTC+8)
4tasks done=6/961 running=2 failed=0 blocked=0 pending=953 | results done=3/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__cora__s42:328s;cuda3:sft:sft__lingu__llama31_4b__cora__s42:322s | GPU cuda2:8821/24564MiB:100%;cuda3:11200/24564MiB:100%260813-22:51:57 (UTC+8)
3tasks done=6/961 running=2 failed=0 blocked=0 pending=953 | results done=3/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__cora__s42:144s;cuda3:sft:sft__lingu__llama31_4b__cora__s42:139s | GPU cuda2:8821/24564MiB:97%;cuda3:11200/24564MiB:100%260813-22:48:56 (UTC+8)
2tasks done=6/961 running=2 failed=0 blocked=0 pending=953 | results done=3/120 | RUNNING cuda2:sft:sft__trace__llama31_4b__cora__s42:6s;cuda3:sft:sft__lingu__llama31_4b__cora__s42:1s | GPU cuda2:1275/24564MiB:15%;cuda3:2765/24564MiB:95%260813-22:46:36 (UTC+8)
1tasks done=0/961 running=0 failed=0 blocked=0 pending=961 | results done=0/120 | RUNNING none | GPU cuda2:887/24564MiB:14%;cuda3:2765/24564MiB:97%260813-22:45:53 (UTC+8)
#06

615TRACE-KDfullexperiment

▦ table
last update
260813-21:13:00
records
2000
type
text
latest
tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:2805s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:1701s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:4034s | GPU cuda1:8699/24564MiB:14%;cuda2:5453/24564MiB:54%;cuda3:10370/24564MiB:100%
ID数值接收时间
2338tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:2805s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:1701s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:4034s | GPU cuda1:8699/24564MiB:14%;cuda2:5453/24564MiB:54%;cuda3:10370/24564MiB:100%260813-21:13:00 (UTC+8)
2337tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:2623s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:1518s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:3852s | GPU cuda1:8699/24564MiB:14%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-21:09:59 (UTC+8)
2336tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:2442s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:1337s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:3671s | GPU cuda1:8699/24564MiB:97%;cuda2:5453/24564MiB:15%;cuda3:10370/24564MiB:100%260813-21:06:56 (UTC+8)
2335tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:2261s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:1156s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:3490s | GPU cuda1:8699/24564MiB:98%;cuda2:5453/24564MiB:15%;cuda3:10370/24564MiB:100%260813-21:03:55 (UTC+8)
2334tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:2080s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:976s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:3309s | GPU cuda1:8699/24564MiB:99%;cuda2:5453/24564MiB:14%;cuda3:10370/24564MiB:100%260813-21:00:54 (UTC+8)
2333tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:1899s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:795s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:3128s | GPU cuda1:8699/24564MiB:98%;cuda2:5453/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:57:53 (UTC+8)
2332tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:1716s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:611s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:2945s | GPU cuda1:8699/24564MiB:99%;cuda2:5453/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:54:53 (UTC+8)
2331tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:1535s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:430s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:2764s | GPU cuda1:8699/24564MiB:98%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-20:51:49 (UTC+8)
2330tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:1354s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:249s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:2582s | GPU cuda1:8699/24564MiB:14%;cuda2:5453/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:48:48 (UTC+8)
2329tasks total=14270 running=3 done=3452 failed=0 blocked=0 pending=10815 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:1173s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__pubmed__s42:68s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:2401s | GPU cuda1:8699/24564MiB:14%;cuda2:5453/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:45:47 (UTC+8)
2328tasks total=14270 running=3 done=3450 failed=0 blocked=0 pending=10817 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:992s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:2830s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:2220s | GPU cuda1:8699/24564MiB:100%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:42:46 (UTC+8)
2327tasks total=14270 running=3 done=3450 failed=0 blocked=0 pending=10817 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:809s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:2648s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:2038s | GPU cuda1:8699/24564MiB:14%;cuda2:5433/24564MiB:16%;cuda3:10370/24564MiB:100%260813-20:39:45 (UTC+8)
2326tasks total=14270 running=3 done=3450 failed=0 blocked=0 pending=10817 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:628s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:2467s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:1857s | GPU cuda1:8699/24564MiB:15%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:36:43 (UTC+8)
2325tasks total=14270 running=3 done=3450 failed=0 blocked=0 pending=10817 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:447s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:2286s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:1676s | GPU cuda1:8699/24564MiB:15%;cuda2:5433/24564MiB:16%;cuda3:10370/24564MiB:100%260813-20:33:42 (UTC+8)
2324tasks total=14270 running=3 done=3450 failed=0 blocked=0 pending=10817 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:266s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:2105s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:1495s | GPU cuda1:8699/24564MiB:98%;cuda2:5431/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:30:40 (UTC+8)
2323tasks total=14270 running=3 done=3450 failed=0 blocked=0 pending=10817 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s3407:85s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:1924s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:1314s | GPU cuda1:8699/24564MiB:98%;cuda2:5431/24564MiB:99%;cuda3:10370/24564MiB:100%260813-20:27:40 (UTC+8)
2322tasks total=14270 running=3 done=3449 failed=0 blocked=0 pending=10818 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:3202s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:1743s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:1133s | GPU cuda1:8699/24564MiB:98%;cuda2:5431/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:24:38 (UTC+8)
2321tasks total=14270 running=3 done=3449 failed=0 blocked=0 pending=10818 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:3021s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:1562s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:952s | GPU cuda1:8699/24564MiB:99%;cuda2:5431/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:21:38 (UTC+8)
2320tasks total=14270 running=3 done=3449 failed=0 blocked=0 pending=10818 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:2840s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:1381s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:771s | GPU cuda1:8699/24564MiB:98%;cuda2:5431/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:18:37 (UTC+8)
2319tasks total=14270 running=3 done=3449 failed=0 blocked=0 pending=10818 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:2659s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:1200s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:590s | GPU cuda1:8699/24564MiB:98%;cuda2:5431/24564MiB:16%;cuda3:10370/24564MiB:100%260813-20:15:35 (UTC+8)
2318tasks total=14270 running=3 done=3449 failed=0 blocked=0 pending=10818 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:2478s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:1019s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:409s | GPU cuda1:8699/24564MiB:98%;cuda2:5119/24564MiB:97%;cuda3:10370/24564MiB:100%260813-20:12:34 (UTC+8)
2317tasks total=14270 running=3 done=3449 failed=0 blocked=0 pending=10818 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:2297s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:838s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:228s | GPU cuda1:8699/24564MiB:98%;cuda2:5119/24564MiB:0%;cuda3:10370/24564MiB:100%260813-20:09:33 (UTC+8)
2316tasks total=14270 running=3 done=3449 failed=0 blocked=0 pending=10818 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:2116s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:657s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__pubmed__s42:47s | GPU cuda1:8475/24564MiB:42%;cuda2:5433/24564MiB:97%;cuda3:10370/24564MiB:100%260813-20:06:32 (UTC+8)
2315tasks total=14270 running=3 done=3448 failed=0 blocked=0 pending=10819 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:1935s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:476s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:5713s | GPU cuda1:8475/24564MiB:98%;cuda2:5433/24564MiB:16%;cuda3:10370/24564MiB:100%260813-20:03:32 (UTC+8)
2314tasks total=14270 running=3 done=3448 failed=0 blocked=0 pending=10819 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:1753s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:294s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:5531s | GPU cuda1:8475/24564MiB:15%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-20:00:31 (UTC+8)
2313tasks total=14270 running=3 done=3448 failed=0 blocked=0 pending=10819 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:1572s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__pubmed__s42:113s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:5349s | GPU cuda1:8475/24564MiB:15%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-19:57:29 (UTC+8)
2312tasks total=14270 running=3 done=3447 failed=0 blocked=0 pending=10820 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:1391s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:2888s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:5169s | GPU cuda1:8475/24564MiB:14%;cuda2:5433/24564MiB:36%;cuda3:10370/24564MiB:100%260813-19:54:28 (UTC+8)
2311tasks total=14270 running=3 done=3447 failed=0 blocked=0 pending=10820 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:1210s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:2707s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:4988s | GPU cuda1:8475/24564MiB:14%;cuda2:5433/24564MiB:16%;cuda3:10370/24564MiB:100%260813-19:51:27 (UTC+8)
2310tasks total=14270 running=3 done=3447 failed=0 blocked=0 pending=10820 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:1029s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:2526s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:4807s | GPU cuda1:8475/24564MiB:13%;cuda2:5433/24564MiB:99%;cuda3:10370/24564MiB:100%260813-19:48:26 (UTC+8)
2309tasks total=14270 running=3 done=3447 failed=0 blocked=0 pending=10820 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:846s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:2342s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:4623s | GPU cuda1:8475/24564MiB:99%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-19:45:25 (UTC+8)
2308tasks total=14270 running=3 done=3447 failed=0 blocked=0 pending=10820 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:664s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:2161s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:4441s | GPU cuda1:8475/24564MiB:99%;cuda2:5433/24564MiB:16%;cuda3:10370/24564MiB:100%260813-19:42:21 (UTC+8)
2307tasks total=14270 running=3 done=3447 failed=0 blocked=0 pending=10820 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:483s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:1980s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:4261s | GPU cuda1:8475/24564MiB:98%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:97%260813-19:39:19 (UTC+8)
2306tasks total=14270 running=3 done=3447 failed=0 blocked=0 pending=10820 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:302s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:1799s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:4080s | GPU cuda1:8475/24564MiB:99%;cuda2:5433/24564MiB:16%;cuda3:10370/24564MiB:100%260813-19:36:19 (UTC+8)
2305tasks total=14270 running=3 done=3447 failed=0 blocked=0 pending=10820 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s3407:121s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:1618s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:3898s | GPU cuda1:8475/24564MiB:98%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-19:33:18 (UTC+8)
2304tasks total=14270 running=3 done=3445 failed=0 blocked=0 pending=10822 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:3057s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:1437s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:3718s | GPU cuda1:8475/24564MiB:99%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-19:30:16 (UTC+8)
2303tasks total=14270 running=3 done=3445 failed=0 blocked=0 pending=10822 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:2876s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:1256s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:3537s | GPU cuda1:8475/24564MiB:14%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-19:27:16 (UTC+8)
2302tasks total=14270 running=3 done=3445 failed=0 blocked=0 pending=10822 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:2695s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:1075s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:3356s | GPU cuda1:8475/24564MiB:12%;cuda2:5433/24564MiB:16%;cuda3:10370/24564MiB:100%260813-19:24:15 (UTC+8)
2301tasks total=14270 running=3 done=3445 failed=0 blocked=0 pending=10822 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:2514s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:894s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:3175s | GPU cuda1:8475/24564MiB:98%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-19:21:14 (UTC+8)
2300tasks total=14270 running=3 done=3445 failed=0 blocked=0 pending=10822 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:2334s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:713s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:2994s | GPU cuda1:8475/24564MiB:14%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-19:18:13 (UTC+8)
2299tasks total=14270 running=3 done=3445 failed=0 blocked=0 pending=10822 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:2152s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:532s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:2812s | GPU cuda1:8475/24564MiB:14%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-19:15:12 (UTC+8)
2298tasks total=14270 running=3 done=3445 failed=0 blocked=0 pending=10822 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:1971s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:351s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:2631s | GPU cuda1:8475/24564MiB:98%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-19:12:10 (UTC+8)
2297tasks total=14270 running=3 done=3445 failed=0 blocked=0 pending=10822 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:1790s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__pubmed__s42:170s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:2451s | GPU cuda1:8475/24564MiB:14%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-19:09:09 (UTC+8)
2296tasks total=14270 running=3 done=3443 failed=0 blocked=0 pending=10824 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:1609s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:2877s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:2270s | GPU cuda1:8475/24564MiB:99%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-19:06:09 (UTC+8)
2295tasks total=14270 running=3 done=3443 failed=0 blocked=0 pending=10824 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:1428s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:2696s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:2088s | GPU cuda1:8475/24564MiB:99%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-19:03:08 (UTC+8)
2294tasks total=14270 running=3 done=3443 failed=0 blocked=0 pending=10824 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:1247s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:2515s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:1907s | GPU cuda1:8475/24564MiB:14%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-19:00:06 (UTC+8)
2293tasks total=14270 running=3 done=3443 failed=0 blocked=0 pending=10824 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:1066s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:2334s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:1726s | GPU cuda1:8475/24564MiB:13%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:57:05 (UTC+8)
2292tasks total=14270 running=3 done=3443 failed=0 blocked=0 pending=10824 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:885s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:2153s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:1546s | GPU cuda1:8475/24564MiB:99%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-18:54:04 (UTC+8)
2291tasks total=14270 running=3 done=3443 failed=0 blocked=0 pending=10824 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:704s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:1972s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:1365s | GPU cuda1:8475/24564MiB:13%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:51:03 (UTC+8)
2290tasks total=14270 running=3 done=3443 failed=0 blocked=0 pending=10824 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:523s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:1791s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:1183s | GPU cuda1:8475/24564MiB:99%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-18:48:03 (UTC+8)
2289tasks total=14270 running=3 done=3443 failed=0 blocked=0 pending=10824 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:342s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:1610s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:1003s | GPU cuda1:8475/24564MiB:15%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-18:45:01 (UTC+8)
2288tasks total=14270 running=3 done=3443 failed=0 blocked=0 pending=10824 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s3407:158s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:1427s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:819s | GPU cuda1:8475/24564MiB:99%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:42:01 (UTC+8)
2287tasks total=14270 running=3 done=3442 failed=0 blocked=0 pending=10825 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:3067s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:1246s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:638s | GPU cuda1:8475/24564MiB:99%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-18:38:57 (UTC+8)
2286tasks total=14270 running=3 done=3442 failed=0 blocked=0 pending=10825 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:2887s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:1065s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:457s | GPU cuda1:8475/24564MiB:13%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:35:56 (UTC+8)
2285tasks total=14270 running=3 done=3442 failed=0 blocked=0 pending=10825 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:2706s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:884s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:276s | GPU cuda1:8473/24564MiB:13%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:32:55 (UTC+8)
2284tasks total=14270 running=3 done=3442 failed=0 blocked=0 pending=10825 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:2525s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:703s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__pubmed__s42:96s | GPU cuda1:8473/24564MiB:99%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:29:54 (UTC+8)
2283tasks total=14270 running=3 done=3439 failed=0 blocked=0 pending=10828 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:2344s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:522s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:2297s | GPU cuda1:8473/24564MiB:14%;cuda2:6293/24564MiB:95%;cuda3:10370/24564MiB:100%260813-18:26:54 (UTC+8)
2282tasks total=14270 running=3 done=3439 failed=0 blocked=0 pending=10828 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:2163s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:341s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:2116s | GPU cuda1:8383/24564MiB:0%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:23:52 (UTC+8)
2281tasks total=14270 running=3 done=3439 failed=0 blocked=0 pending=10828 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:1982s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__pubmed__s42:160s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:1935s | GPU cuda1:8469/24564MiB:99%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:20:52 (UTC+8)
2280tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:1801s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:2877s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:1754s | GPU cuda1:8469/24564MiB:15%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:17:51 (UTC+8)
2279tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:1620s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:2695s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:1573s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:14:50 (UTC+8)
2278tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:1438s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:2514s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:1392s | GPU cuda1:8469/24564MiB:14%;cuda2:6293/24564MiB:97%;cuda3:10370/24564MiB:100%260813-18:11:48 (UTC+8)
2277tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:1257s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:2332s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:1210s | GPU cuda1:8469/24564MiB:97%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-18:08:47 (UTC+8)
2276tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:1076s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:2152s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:1029s | GPU cuda1:8469/24564MiB:14%;cuda2:6293/24564MiB:14%;cuda3:10370/24564MiB:99%260813-18:05:46 (UTC+8)
2275tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:895s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:1971s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:848s | GPU cuda1:8469/24564MiB:13%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-18:02:45 (UTC+8)
2274tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:714s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:1790s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:667s | GPU cuda1:8469/24564MiB:15%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:59:44 (UTC+8)
2273tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:532s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:1607s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:485s | GPU cuda1:8469/24564MiB:15%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:56:43 (UTC+8)
2272tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:350s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:1426s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:304s | GPU cuda1:8469/24564MiB:15%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:53:40 (UTC+8)
2271tasks total=14270 running=3 done=3438 failed=0 blocked=0 pending=10829 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s3407:170s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:1245s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__pubmed__s42:123s | GPU cuda1:8858/24564MiB:14%;cuda2:6682/24564MiB:96%;cuda3:10370/24564MiB:100%260813-17:50:39 (UTC+8)
2270tasks total=14270 running=3 done=3435 failed=0 blocked=0 pending=10832 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:2653s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:1064s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:2307s | GPU cuda1:8858/24564MiB:97%;cuda2:6682/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:47:38 (UTC+8)
2269tasks total=14270 running=3 done=3435 failed=0 blocked=0 pending=10832 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:2472s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:883s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:2127s | GPU cuda1:8858/24564MiB:97%;cuda2:6682/24564MiB:97%;cuda3:10370/24564MiB:99%260813-17:44:37 (UTC+8)
2268tasks total=14270 running=3 done=3435 failed=0 blocked=0 pending=10832 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:2291s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:702s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:1946s | GPU cuda1:8858/24564MiB:97%;cuda2:6682/24564MiB:96%;cuda3:10370/24564MiB:100%260813-17:41:37 (UTC+8)
2267tasks total=14270 running=3 done=3435 failed=0 blocked=0 pending=10832 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:2110s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:522s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:1765s | GPU cuda1:8858/24564MiB:97%;cuda2:6682/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:38:36 (UTC+8)
2266tasks total=14270 running=3 done=3435 failed=0 blocked=0 pending=10832 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:1929s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:340s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:1583s | GPU cuda1:8858/24564MiB:97%;cuda2:6682/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:35:35 (UTC+8)
2265tasks total=14270 running=3 done=3435 failed=0 blocked=0 pending=10832 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:1748s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__pubmed__s42:159s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:1402s | GPU cuda1:8856/24564MiB:100%;cuda2:6682/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:32:33 (UTC+8)
2264tasks total=14270 running=3 done=3433 failed=0 blocked=0 pending=10834 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:1567s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__pubmed__s42:1621s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:1222s | GPU cuda1:8856/24564MiB:95%;cuda2:6120/24564MiB:98%;cuda3:10370/24564MiB:100%260813-17:29:32 (UTC+8)
2263tasks total=14270 running=3 done=3433 failed=0 blocked=0 pending=10834 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:1386s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__pubmed__s42:1440s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:1040s | GPU cuda1:8856/24564MiB:100%;cuda2:6130/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:26:32 (UTC+8)
2262tasks total=14270 running=3 done=3433 failed=0 blocked=0 pending=10834 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:1205s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__pubmed__s42:1259s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:860s | GPU cuda1:8856/24564MiB:100%;cuda2:6130/24564MiB:91%;cuda3:10370/24564MiB:100%260813-17:23:31 (UTC+8)
2261tasks total=14270 running=3 done=3433 failed=0 blocked=0 pending=10834 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:1021s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__pubmed__s42:1075s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:676s | GPU cuda1:8856/24564MiB:100%;cuda2:6130/24564MiB:98%;cuda3:10370/24564MiB:100%260813-17:20:30 (UTC+8)
2260tasks total=14270 running=3 done=3433 failed=0 blocked=0 pending=10834 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:839s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__pubmed__s42:894s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:494s | GPU cuda1:8469/24564MiB:97%;cuda2:5743/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:17:26 (UTC+8)
2259tasks total=14270 running=3 done=3433 failed=0 blocked=0 pending=10834 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:659s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__pubmed__s42:713s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:313s | GPU cuda1:8469/24564MiB:97%;cuda2:5743/24564MiB:97%;cuda3:10370/24564MiB:100%260813-17:14:24 (UTC+8)
2258tasks total=14270 running=3 done=3433 failed=0 blocked=0 pending=10834 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:477s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__pubmed__s42:531s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__pubmed__s42:132s | GPU cuda1:8469/24564MiB:100%;cuda2:5743/24564MiB:100%;cuda3:10370/24564MiB:100%260813-17:11:23 (UTC+8)
2257tasks total=14270 running=3 done=3430 failed=0 blocked=0 pending=10837 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:296s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__pubmed__s42:350s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:2029s | GPU cuda1:8469/24564MiB:97%;cuda2:5733/24564MiB:98%;cuda3:10370/24564MiB:100%260813-17:08:22 (UTC+8)
2256tasks total=14270 running=3 done=3430 failed=0 blocked=0 pending=10837 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s3407:114s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__pubmed__s42:169s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:1847s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:41%;cuda3:10370/24564MiB:100%260813-17:05:21 (UTC+8)
2255tasks total=14270 running=3 done=3428 failed=0 blocked=0 pending=10839 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:2596s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__pubmed__s42:1616s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:1666s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:15%;cuda3:10370/24564MiB:100%260813-17:02:19 (UTC+8)
2254tasks total=14270 running=3 done=3428 failed=0 blocked=0 pending=10839 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:2415s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__pubmed__s42:1435s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:1485s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-16:59:18 (UTC+8)
2253tasks total=14270 running=3 done=3428 failed=0 blocked=0 pending=10839 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:2235s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__pubmed__s42:1255s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:1304s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-16:56:17 (UTC+8)
2252tasks total=14270 running=3 done=3428 failed=0 blocked=0 pending=10839 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:2054s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__pubmed__s42:1074s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:1123s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:98%;cuda3:10370/24564MiB:100%260813-16:53:16 (UTC+8)
2251tasks total=14270 running=3 done=3428 failed=0 blocked=0 pending=10839 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:1873s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__pubmed__s42:893s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:942s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-16:50:15 (UTC+8)
2250tasks total=14270 running=3 done=3428 failed=0 blocked=0 pending=10839 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:1691s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__pubmed__s42:712s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:761s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-16:47:14 (UTC+8)
2249tasks total=14270 running=3 done=3428 failed=0 blocked=0 pending=10839 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:1510s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__pubmed__s42:530s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:580s | GPU cuda1:8469/24564MiB:97%;cuda2:5733/24564MiB:98%;cuda3:10370/24564MiB:100%260813-16:44:13 (UTC+8)
2248tasks total=14270 running=3 done=3428 failed=0 blocked=0 pending=10839 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:1329s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__pubmed__s42:350s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:399s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:99%260813-16:41:12 (UTC+8)
2247tasks total=14270 running=3 done=3428 failed=0 blocked=0 pending=10839 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:1148s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__pubmed__s42:168s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:218s | GPU cuda1:8469/24564MiB:97%;cuda2:5733/24564MiB:15%;cuda3:10370/24564MiB:100%260813-16:38:11 (UTC+8)
2246tasks total=14270 running=3 done=3426 failed=0 blocked=0 pending=10841 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:967s;cuda2:trace_context:trace_bank__n1__Semantic-128-b__llama31_4b__pubmed__s42:899s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__pubmed__s42:37s | GPU cuda1:8469/24564MiB:100%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-16:35:10 (UTC+8)
2245tasks total=14270 running=3 done=3425 failed=0 blocked=0 pending=10842 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:786s;cuda2:trace_context:trace_bank__n1__Semantic-128-b__llama31_4b__pubmed__s42:718s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:1925s | GPU cuda1:8469/24564MiB:100%;cuda2:5433/24564MiB:99%;cuda3:10370/24564MiB:100%260813-16:32:09 (UTC+8)
2244tasks total=14270 running=3 done=3425 failed=0 blocked=0 pending=10842 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:605s;cuda2:trace_context:trace_bank__n1__Semantic-128-b__llama31_4b__pubmed__s42:537s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:1744s | GPU cuda1:8469/24564MiB:100%;cuda2:5433/24564MiB:99%;cuda3:10370/24564MiB:100%260813-16:29:08 (UTC+8)
2243tasks total=14270 running=3 done=3425 failed=0 blocked=0 pending=10842 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:233s;cuda2:trace_context:trace_bank__n1__Semantic-128-b__llama31_4b__pubmed__s42:165s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:1373s | GPU cuda1:8469/24564MiB:97%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-16:22:56 (UTC+8)
2242tasks total=14270 running=3 done=3424 failed=0 blocked=0 pending=10843 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s3407:52s;cuda2:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__pubmed__s42:893s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:1192s | GPU cuda1:8469/24564MiB:96%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-16:19:55 (UTC+8)
2241tasks total=14270 running=3 done=3422 failed=0 blocked=0 pending=10845 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s3407:1547s;cuda2:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__pubmed__s42:712s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:1011s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:15%;cuda3:10370/24564MiB:100%260813-16:16:54 (UTC+8)
2240tasks total=14270 running=3 done=3422 failed=0 blocked=0 pending=10845 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s3407:1365s;cuda2:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__pubmed__s42:529s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:828s | GPU cuda1:8469/24564MiB:97%;cuda2:5433/24564MiB:16%;cuda3:10370/24564MiB:100%260813-16:13:53 (UTC+8)
2239tasks total=14270 running=3 done=3422 failed=0 blocked=0 pending=10845 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s3407:1184s;cuda2:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__pubmed__s42:348s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:647s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:99%;cuda3:10370/24564MiB:100%260813-16:10:51 (UTC+8)
2238tasks total=14270 running=3 done=3422 failed=0 blocked=0 pending=10845 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s3407:1003s;cuda2:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__pubmed__s42:168s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:466s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:52%;cuda3:10370/24564MiB:100%260813-16:07:50 (UTC+8)
2237tasks total=14270 running=3 done=3420 failed=0 blocked=0 pending=10847 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s3407:822s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:2856s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:286s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-16:04:49 (UTC+8)
2236tasks total=14270 running=3 done=3420 failed=0 blocked=0 pending=10847 | results done=1142/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s3407:641s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:2675s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__pubmed__s42:105s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-16:01:48 (UTC+8)
2235tasks total=14270 running=3 done=3415 failed=0 blocked=0 pending=10852 | results done=1138/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s3407:460s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:2494s;cuda3:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__pubmed__s42__gcn3:29s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:3614/24564MiB:100%260813-15:58:47 (UTC+8)
2234tasks total=14270 running=3 done=3411 failed=0 blocked=0 pending=10856 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s3407:280s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:2313s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:2298s | GPU cuda1:8469/24564MiB:100%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-15:55:46 (UTC+8)
2233tasks total=14270 running=3 done=3411 failed=0 blocked=0 pending=10856 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s3407:99s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:2133s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:2117s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-15:52:45 (UTC+8)
2232tasks total=14270 running=3 done=3410 failed=0 blocked=0 pending=10857 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s3407:1592s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:1952s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:1936s | GPU cuda1:8469/24564MiB:100%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:49:45 (UTC+8)
2231tasks total=14270 running=3 done=3410 failed=0 blocked=0 pending=10857 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s3407:1220s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:1580s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:1565s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:43:33 (UTC+8)
2230tasks total=14270 running=3 done=3410 failed=0 blocked=0 pending=10857 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s3407:1036s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:1396s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:1380s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:40:32 (UTC+8)
2229tasks total=14270 running=3 done=3410 failed=0 blocked=0 pending=10857 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s3407:855s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:1215s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:1199s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:37:28 (UTC+8)
2228tasks total=14270 running=3 done=3410 failed=0 blocked=0 pending=10857 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s3407:674s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:1034s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:1019s | GPU cuda1:8469/24564MiB:100%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:34:27 (UTC+8)
2227tasks total=14270 running=3 done=3410 failed=0 blocked=0 pending=10857 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s3407:493s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:853s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:837s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-15:31:26 (UTC+8)
2226tasks total=14270 running=3 done=3410 failed=0 blocked=0 pending=10857 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s3407:310s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:670s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:655s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:28:25 (UTC+8)
2225tasks total=14270 running=3 done=3410 failed=0 blocked=0 pending=10857 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s3407:129s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:489s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:474s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:25:22 (UTC+8)
2224tasks total=14270 running=3 done=3408 failed=0 blocked=0 pending=10859 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s3407:955s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:309s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:293s | GPU cuda1:8469/24564MiB:99%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-15:22:21 (UTC+8)
2223tasks total=14270 running=3 done=3408 failed=0 blocked=0 pending=10859 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s3407:774s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__pubmed__s42:127s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__pubmed__s42:112s | GPU cuda1:8469/24564MiB:14%;cuda2:6293/24564MiB:95%;cuda3:10370/24564MiB:100%260813-15:19:21 (UTC+8)
2222tasks total=14270 running=3 done=3406 failed=0 blocked=0 pending=10861 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s3407:593s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:2817s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:2334s | GPU cuda1:8469/24564MiB:14%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:16:19 (UTC+8)
2221tasks total=14270 running=3 done=3406 failed=0 blocked=0 pending=10861 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s3407:411s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:2635s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:2152s | GPU cuda1:8469/24564MiB:97%;cuda2:6293/24564MiB:96%;cuda3:10370/24564MiB:100%260813-15:13:18 (UTC+8)
2220tasks total=14270 running=3 done=3406 failed=0 blocked=0 pending=10861 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s3407:230s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:2455s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:1972s | GPU cuda1:8469/24564MiB:99%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:10:17 (UTC+8)
2219tasks total=14270 running=3 done=3406 failed=0 blocked=0 pending=10861 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s3407:50s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:2274s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:1791s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:07:16 (UTC+8)
2218tasks total=14270 running=3 done=3405 failed=0 blocked=0 pending=10862 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s3407:873s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:2093s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:1610s | GPU cuda1:8469/24564MiB:99%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:04:15 (UTC+8)
2217tasks total=14270 running=3 done=3405 failed=0 blocked=0 pending=10862 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s3407:692s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:1912s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:1429s | GPU cuda1:8469/24564MiB:97%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-15:01:14 (UTC+8)
2216tasks total=14270 running=3 done=3405 failed=0 blocked=0 pending=10862 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s3407:510s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:1731s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:1248s | GPU cuda1:8469/24564MiB:99%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:99%260813-14:58:13 (UTC+8)
2215tasks total=14270 running=3 done=3405 failed=0 blocked=0 pending=10862 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s3407:330s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:1550s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:1067s | GPU cuda1:8469/24564MiB:99%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:55:12 (UTC+8)
2214tasks total=14270 running=3 done=3405 failed=0 blocked=0 pending=10862 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s3407:149s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:1369s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:886s | GPU cuda1:8469/24564MiB:98%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:52:11 (UTC+8)
2213tasks total=14270 running=3 done=3403 failed=0 blocked=0 pending=10864 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:2604s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:1187s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:704s | GPU cuda1:8469/24564MiB:100%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:49:10 (UTC+8)
2212tasks total=14270 running=3 done=3403 failed=0 blocked=0 pending=10864 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:2423s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:1006s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:523s | GPU cuda1:8469/24564MiB:97%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:96%260813-14:46:08 (UTC+8)
2211tasks total=14270 running=3 done=3403 failed=0 blocked=0 pending=10864 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:2240s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:823s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:340s | GPU cuda1:8469/24564MiB:100%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:43:07 (UTC+8)
2210tasks total=14270 running=3 done=3403 failed=0 blocked=0 pending=10864 | results done=1136/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:2060s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:642s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__pubmed__s42:159s | GPU cuda1:8469/24564MiB:97%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:40:04 (UTC+8)
2209tasks total=14270 running=3 done=3401 failed=0 blocked=0 pending=10866 | results done=1135/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:1879s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:461s;cuda3:fixed_downstream:run__T1_FixedFull512__llama3_8b__pubmed__s42__sage3:6s | GPU cuda1:8469/24564MiB:97%;cuda2:6293/24564MiB:100%;cuda3:3516/24564MiB:100%260813-14:37:04 (UTC+8)
2208tasks total=14270 running=3 done=3394 failed=0 blocked=0 pending=10873 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:1698s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:281s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:3369s | GPU cuda1:8469/24564MiB:96%;cuda2:6293/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:34:03 (UTC+8)
2207tasks total=14270 running=3 done=3394 failed=0 blocked=0 pending=10873 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:1517s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__pubmed__s42:100s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:3188s | GPU cuda1:8469/24564MiB:97%;cuda2:6293/24564MiB:98%;cuda3:10370/24564MiB:100%260813-14:31:02 (UTC+8)
2206tasks total=14270 running=3 done=3392 failed=0 blocked=0 pending=10875 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:1336s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__pubmed__s42:1520s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:3007s | GPU cuda1:8469/24564MiB:96%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:98%260813-14:28:01 (UTC+8)
2205tasks total=14270 running=3 done=3392 failed=0 blocked=0 pending=10875 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:1155s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__pubmed__s42:1339s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:2826s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:25:00 (UTC+8)
2204tasks total=14270 running=3 done=3392 failed=0 blocked=0 pending=10875 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:974s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__pubmed__s42:1158s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:2645s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:97%;cuda3:10370/24564MiB:100%260813-14:21:59 (UTC+8)
2203tasks total=14270 running=3 done=3392 failed=0 blocked=0 pending=10875 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:793s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__pubmed__s42:977s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:2464s | GPU cuda1:8469/24564MiB:97%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:18:58 (UTC+8)
2202tasks total=14270 running=3 done=3392 failed=0 blocked=0 pending=10875 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:612s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__pubmed__s42:795s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:2283s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:97%;cuda3:10370/24564MiB:100%260813-14:15:57 (UTC+8)
2201tasks total=14270 running=3 done=3392 failed=0 blocked=0 pending=10875 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:429s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__pubmed__s42:613s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:2100s | GPU cuda1:8469/24564MiB:97%;cuda2:5733/24564MiB:98%;cuda3:10370/24564MiB:100%260813-14:12:56 (UTC+8)
2200tasks total=14270 running=3 done=3392 failed=0 blocked=0 pending=10875 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:247s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__pubmed__s42:431s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:1918s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:09:53 (UTC+8)
2199tasks total=14270 running=3 done=3392 failed=0 blocked=0 pending=10875 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s3407:66s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__pubmed__s42:250s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:1737s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:98%;cuda3:10370/24564MiB:100%260813-14:06:51 (UTC+8)
2198tasks total=14270 running=3 done=3391 failed=0 blocked=0 pending=10876 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:2523s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__pubmed__s42:69s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:1556s | GPU cuda1:8469/24564MiB:97%;cuda2:5733/24564MiB:97%;cuda3:10370/24564MiB:100%260813-14:03:50 (UTC+8)
2197tasks total=14270 running=3 done=3390 failed=0 blocked=0 pending=10877 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:2342s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__pubmed__s42:1496s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:1375s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-14:00:49 (UTC+8)
2196tasks total=14270 running=3 done=3390 failed=0 blocked=0 pending=10877 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:2161s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__pubmed__s42:1315s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:1194s | GPU cuda1:8469/24564MiB:97%;cuda2:5733/24564MiB:97%;cuda3:10370/24564MiB:100%260813-13:57:49 (UTC+8)
2195tasks total=14270 running=3 done=3390 failed=0 blocked=0 pending=10877 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:1980s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__pubmed__s42:1134s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:1013s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-13:54:47 (UTC+8)
2194tasks total=14270 running=3 done=3390 failed=0 blocked=0 pending=10877 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:1798s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__pubmed__s42:952s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:831s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:51:46 (UTC+8)
2193tasks total=14270 running=3 done=3390 failed=0 blocked=0 pending=10877 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:1617s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__pubmed__s42:771s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:650s | GPU cuda1:8469/24564MiB:97%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-13:48:44 (UTC+8)
2192tasks total=14270 running=3 done=3390 failed=0 blocked=0 pending=10877 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:1436s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__pubmed__s42:590s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:469s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:97%;cuda3:10370/24564MiB:100%260813-13:45:43 (UTC+8)
2191tasks total=14270 running=3 done=3390 failed=0 blocked=0 pending=10877 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:1253s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__pubmed__s42:407s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:286s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:42:42 (UTC+8)
2190tasks total=14270 running=3 done=3390 failed=0 blocked=0 pending=10877 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:1072s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__pubmed__s42:226s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__pubmed__s42:105s | GPU cuda1:8469/24564MiB:100%;cuda2:5733/24564MiB:100%;cuda3:10370/24564MiB:100%260813-13:39:39 (UTC+8)
2189tasks total=14270 running=3 done=3389 failed=0 blocked=0 pending=10878 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:891s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__pubmed__s42:45s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:3338s | GPU cuda1:8469/24564MiB:97%;cuda2:5733/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:36:38 (UTC+8)
2188tasks total=14270 running=3 done=3387 failed=0 blocked=0 pending=10880 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:710s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama31_4b__pubmed__s42:996s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:3157s | GPU cuda1:8469/24564MiB:100%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:33:37 (UTC+8)
2187tasks total=14270 running=3 done=3387 failed=0 blocked=0 pending=10880 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:529s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama31_4b__pubmed__s42:815s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:2976s | GPU cuda1:8469/24564MiB:100%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:30:36 (UTC+8)
2186tasks total=14270 running=3 done=3387 failed=0 blocked=0 pending=10880 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:347s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama31_4b__pubmed__s42:633s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:2794s | GPU cuda1:8469/24564MiB:100%;cuda2:5453/24564MiB:97%;cuda3:10370/24564MiB:100%260813-13:27:35 (UTC+8)
2185tasks total=14270 running=3 done=3387 failed=0 blocked=0 pending=10880 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s3407:166s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama31_4b__pubmed__s42:452s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:2613s | GPU cuda1:8469/24564MiB:100%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:24:33 (UTC+8)
2184tasks total=14270 running=3 done=3385 failed=0 blocked=0 pending=10882 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:1635s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama31_4b__pubmed__s42:271s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:2432s | GPU cuda1:8469/24564MiB:98%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:21:32 (UTC+8)
2183tasks total=14270 running=3 done=3385 failed=0 blocked=0 pending=10882 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:1454s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama31_4b__pubmed__s42:90s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:2251s | GPU cuda1:8469/24564MiB:100%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:18:31 (UTC+8)
2182tasks total=14270 running=3 done=3384 failed=0 blocked=0 pending=10883 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:1273s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama31_4b__pubmed__s42:1039s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:2070s | GPU cuda1:8469/24564MiB:100%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:15:30 (UTC+8)
2181tasks total=14270 running=3 done=3384 failed=0 blocked=0 pending=10883 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:1091s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama31_4b__pubmed__s42:857s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:1889s | GPU cuda1:8469/24564MiB:98%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:12:29 (UTC+8)
2180tasks total=14270 running=3 done=3384 failed=0 blocked=0 pending=10883 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:909s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama31_4b__pubmed__s42:675s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:1707s | GPU cuda1:8469/24564MiB:98%;cuda2:5453/24564MiB:27%;cuda3:10370/24564MiB:100%260813-13:09:28 (UTC+8)
2179tasks total=14270 running=3 done=3384 failed=0 blocked=0 pending=10883 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:729s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama31_4b__pubmed__s42:494s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:1526s | GPU cuda1:8469/24564MiB:100%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:06:26 (UTC+8)
2178tasks total=14270 running=3 done=3384 failed=0 blocked=0 pending=10883 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:548s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama31_4b__pubmed__s42:314s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:1345s | GPU cuda1:8469/24564MiB:98%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:100%260813-13:03:25 (UTC+8)
2177tasks total=14270 running=3 done=3384 failed=0 blocked=0 pending=10883 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:367s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama31_4b__pubmed__s42:133s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:1164s | GPU cuda1:8469/24564MiB:98%;cuda2:5453/24564MiB:98%;cuda3:10370/24564MiB:99%260813-13:00:24 (UTC+8)
2176tasks total=14270 running=3 done=3382 failed=0 blocked=0 pending=10885 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:184s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama31_4b__pubmed__s42:833s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:982s | GPU cuda1:8469/24564MiB:100%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-12:57:23 (UTC+8)
2175tasks total=14270 running=3 done=3382 failed=0 blocked=0 pending=10885 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s3407:3s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama31_4b__pubmed__s42:652s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:801s | GPU cuda1:1019/24564MiB:14%;cuda2:5433/24564MiB:86%;cuda3:10370/24564MiB:100%260813-12:54:21 (UTC+8)
2174tasks total=14270 running=3 done=3381 failed=0 blocked=0 pending=10886 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s3407:1470s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama31_4b__pubmed__s42:471s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:619s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-12:51:20 (UTC+8)
2173tasks total=14270 running=3 done=3381 failed=0 blocked=0 pending=10886 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s3407:1289s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama31_4b__pubmed__s42:290s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:439s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-12:48:19 (UTC+8)
2172tasks total=14270 running=3 done=3381 failed=0 blocked=0 pending=10886 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s3407:1107s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama31_4b__pubmed__s42:108s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:257s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:99%;cuda3:10370/24564MiB:100%260813-12:45:18 (UTC+8)
2171tasks total=14270 running=3 done=3380 failed=0 blocked=0 pending=10887 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s3407:926s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama31_4b__pubmed__s42:807s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__pubmed__s42:75s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:97%;cuda3:10370/24564MiB:100%260813-12:42:16 (UTC+8)
2170tasks total=14270 running=3 done=3379 failed=0 blocked=0 pending=10888 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s3407:745s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama31_4b__pubmed__s42:626s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:1767s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-12:39:14 (UTC+8)
2169tasks total=14270 running=3 done=3379 failed=0 blocked=0 pending=10888 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s3407:564s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama31_4b__pubmed__s42:445s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:1586s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-12:36:13 (UTC+8)
2168tasks total=14270 running=3 done=3379 failed=0 blocked=0 pending=10888 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s3407:383s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama31_4b__pubmed__s42:264s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:1405s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-12:33:12 (UTC+8)
2167tasks total=14270 running=3 done=3379 failed=0 blocked=0 pending=10888 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s3407:202s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama31_4b__pubmed__s42:83s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:1224s | GPU cuda1:8469/24564MiB:98%;cuda2:5433/24564MiB:98%;cuda3:10370/24564MiB:100%260813-12:30:11 (UTC+8)
2166tasks total=14270 running=3 done=3377 failed=0 blocked=0 pending=10890 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s3407:20s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:2783s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:1042s | GPU cuda1:8337/24564MiB:36%;cuda2:6291/24564MiB:100%;cuda3:10370/24564MiB:100%260813-12:27:10 (UTC+8)
2165tasks total=14270 running=3 done=3375 failed=0 blocked=0 pending=10892 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s3407:1037s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:2602s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:861s | GPU cuda1:8469/24564MiB:98%;cuda2:6291/24564MiB:96%;cuda3:10370/24564MiB:100%260813-12:24:08 (UTC+8)
2164tasks total=14270 running=3 done=3375 failed=0 blocked=0 pending=10892 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s3407:856s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:2421s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:680s | GPU cuda1:8469/24564MiB:97%;cuda2:6291/24564MiB:100%;cuda3:10370/24564MiB:100%260813-12:21:08 (UTC+8)
2163tasks total=14270 running=3 done=3375 failed=0 blocked=0 pending=10892 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s3407:675s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:2240s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:499s | GPU cuda1:8469/24564MiB:98%;cuda2:6291/24564MiB:100%;cuda3:10370/24564MiB:100%260813-12:18:07 (UTC+8)
2162tasks total=14270 running=3 done=3375 failed=0 blocked=0 pending=10892 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s3407:494s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:2059s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:318s | GPU cuda1:8469/24564MiB:98%;cuda2:6291/24564MiB:96%;cuda3:10370/24564MiB:100%260813-12:15:05 (UTC+8)
2161tasks total=14270 running=3 done=3375 failed=0 blocked=0 pending=10892 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s3407:313s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:1878s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__pubmed__s42:137s | GPU cuda1:8469/24564MiB:99%;cuda2:6291/24564MiB:96%;cuda3:10370/24564MiB:100%260813-12:12:04 (UTC+8)
2160tasks total=14270 running=3 done=3374 failed=0 blocked=0 pending=10893 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s3407:132s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:1697s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:7142s | GPU cuda1:8469/24564MiB:98%;cuda2:6291/24564MiB:100%;cuda3:12996/24564MiB:100%260813-12:09:03 (UTC+8)
2159tasks total=14270 running=3 done=3373 failed=0 blocked=0 pending=10894 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s3407:1146s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:1515s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:6961s | GPU cuda1:8469/24564MiB:98%;cuda2:6291/24564MiB:96%;cuda3:12996/24564MiB:100%260813-12:06:03 (UTC+8)
2158tasks total=14270 running=3 done=3373 failed=0 blocked=0 pending=10894 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s3407:965s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:1334s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:6780s | GPU cuda1:8469/24564MiB:98%;cuda2:6291/24564MiB:100%;cuda3:12996/24564MiB:100%260813-12:03:01 (UTC+8)
2157tasks total=14270 running=3 done=3373 failed=0 blocked=0 pending=10894 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s3407:784s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:1153s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:6598s | GPU cuda1:8469/24564MiB:98%;cuda2:6291/24564MiB:96%;cuda3:12996/24564MiB:100%260813-12:00:00 (UTC+8)
2156tasks total=14270 running=3 done=3373 failed=0 blocked=0 pending=10894 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s3407:603s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:972s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:6418s | GPU cuda1:8469/24564MiB:99%;cuda2:6291/24564MiB:100%;cuda3:12996/24564MiB:100%260813-11:56:59 (UTC+8)
2155tasks total=14270 running=3 done=3373 failed=0 blocked=0 pending=10894 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s3407:422s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:791s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:6237s | GPU cuda1:8469/24564MiB:98%;cuda2:6291/24564MiB:100%;cuda3:12996/24564MiB:100%260813-11:53:58 (UTC+8)
2154tasks total=14270 running=3 done=3373 failed=0 blocked=0 pending=10894 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s3407:241s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:610s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:6055s | GPU cuda1:8469/24564MiB:98%;cuda2:6291/24564MiB:96%;cuda3:12996/24564MiB:100%260813-11:50:57 (UTC+8)
2153tasks total=14270 running=3 done=3373 failed=0 blocked=0 pending=10894 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s3407:59s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:429s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:5874s | GPU cuda1:8467/24564MiB:98%;cuda2:6291/24564MiB:96%;cuda3:12996/24564MiB:100%260813-11:47:56 (UTC+8)
2152tasks total=14270 running=3 done=3371 failed=0 blocked=0 pending=10896 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s3407:861s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:248s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:5693s | GPU cuda1:8467/24564MiB:99%;cuda2:6291/24564MiB:100%;cuda3:12996/24564MiB:100%260813-11:44:55 (UTC+8)
2151tasks total=14270 running=3 done=3371 failed=0 blocked=0 pending=10896 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s3407:680s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__pubmed__s42:67s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:5512s | GPU cuda1:8467/24564MiB:99%;cuda2:6291/24564MiB:100%;cuda3:12996/24564MiB:100%260813-11:41:54 (UTC+8)
2150tasks total=14270 running=3 done=3370 failed=0 blocked=0 pending=10897 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s3407:500s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:2777s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:5332s | GPU cuda1:8467/24564MiB:99%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-11:38:53 (UTC+8)
2149tasks total=14270 running=3 done=3370 failed=0 blocked=0 pending=10897 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s3407:318s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:2596s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:5150s | GPU cuda1:8467/24564MiB:84%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-11:35:52 (UTC+8)
2148tasks total=14270 running=3 done=3370 failed=0 blocked=0 pending=10897 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s3407:133s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:2410s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:4965s | GPU cuda1:8467/24564MiB:99%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-11:32:51 (UTC+8)
2147tasks total=14270 running=3 done=3369 failed=0 blocked=0 pending=10898 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s3407:932s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:2229s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:4783s | GPU cuda1:8467/24564MiB:99%;cuda2:6291/24564MiB:96%;cuda3:11093/24564MiB:100%260813-11:29:45 (UTC+8)
2146tasks total=14270 running=3 done=3369 failed=0 blocked=0 pending=10898 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s3407:751s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:2048s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:4602s | GPU cuda1:8467/24564MiB:98%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-11:26:44 (UTC+8)
2145tasks total=14270 running=3 done=3369 failed=0 blocked=0 pending=10898 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s3407:569s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:1866s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:4421s | GPU cuda1:8467/24564MiB:99%;cuda2:6291/24564MiB:96%;cuda3:11093/24564MiB:100%260813-11:23:42 (UTC+8)
2144tasks total=14270 running=3 done=3369 failed=0 blocked=0 pending=10898 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s3407:388s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:1685s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:4239s | GPU cuda1:8467/24564MiB:99%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-11:20:41 (UTC+8)
2143tasks total=14270 running=3 done=3369 failed=0 blocked=0 pending=10898 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s3407:207s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:1504s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:4058s | GPU cuda1:8467/24564MiB:97%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-11:17:40 (UTC+8)
2142tasks total=14270 running=3 done=3369 failed=0 blocked=0 pending=10898 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s3407:26s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:1323s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:3878s | GPU cuda1:8467/24564MiB:99%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-11:14:39 (UTC+8)
2141tasks total=14270 running=3 done=3367 failed=0 blocked=0 pending=10900 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:2398s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:1142s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:3696s | GPU cuda1:8467/24564MiB:100%;cuda2:6291/24564MiB:96%;cuda3:11093/24564MiB:100%260813-11:11:38 (UTC+8)
2140tasks total=14270 running=3 done=3367 failed=0 blocked=0 pending=10900 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:2217s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:961s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:3516s | GPU cuda1:8467/24564MiB:97%;cuda2:6291/24564MiB:96%;cuda3:11093/24564MiB:100%260813-11:08:37 (UTC+8)
2139tasks total=14270 running=3 done=3367 failed=0 blocked=0 pending=10900 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:2035s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:778s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:3333s | GPU cuda1:8467/24564MiB:100%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-11:05:36 (UTC+8)
2138tasks total=14270 running=3 done=3367 failed=0 blocked=0 pending=10900 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:1854s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:598s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:3152s | GPU cuda1:8467/24564MiB:97%;cuda2:6291/24564MiB:96%;cuda3:11093/24564MiB:100%260813-11:02:33 (UTC+8)
2137tasks total=14270 running=3 done=3367 failed=0 blocked=0 pending=10900 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:1672s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:416s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:2971s | GPU cuda1:8467/24564MiB:100%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-10:59:32 (UTC+8)
2136tasks total=14270 running=3 done=3367 failed=0 blocked=0 pending=10900 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:1491s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:235s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:2789s | GPU cuda1:8467/24564MiB:100%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-10:56:31 (UTC+8)
2135tasks total=14270 running=3 done=3367 failed=0 blocked=0 pending=10900 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:1310s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__pubmed__s42:54s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:2608s | GPU cuda1:8467/24564MiB:100%;cuda2:6291/24564MiB:100%;cuda3:11093/24564MiB:100%260813-10:53:30 (UTC+8)
2134tasks total=14270 running=3 done=3365 failed=0 blocked=0 pending=10902 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:1129s;cuda2:trace_context:trace_bank__n1__Full-256-b__llama31_4b__pubmed__s42:926s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:2427s | GPU cuda1:8467/24564MiB:100%;cuda2:5451/24564MiB:98%;cuda3:11091/24564MiB:100%260813-10:50:29 (UTC+8)
2133tasks total=14270 running=3 done=3365 failed=0 blocked=0 pending=10902 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:948s;cuda2:trace_context:trace_bank__n1__Full-256-b__llama31_4b__pubmed__s42:745s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:2246s | GPU cuda1:8465/24564MiB:100%;cuda2:4973/24564MiB:97%;cuda3:11091/24564MiB:100%260813-10:47:28 (UTC+8)
2132tasks total=14270 running=3 done=3365 failed=0 blocked=0 pending=10902 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:767s;cuda2:trace_context:trace_bank__n1__Full-256-b__llama31_4b__pubmed__s42:564s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:2065s | GPU cuda1:8465/24564MiB:100%;cuda2:4983/24564MiB:97%;cuda3:11091/24564MiB:100%260813-10:44:27 (UTC+8)
2131tasks total=14270 running=3 done=3365 failed=0 blocked=0 pending=10902 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:585s;cuda2:trace_context:trace_bank__n1__Full-256-b__llama31_4b__pubmed__s42:382s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:1883s | GPU cuda1:7851/24564MiB:95%;cuda2:4595/24564MiB:43%;cuda3:10477/24564MiB:100%260813-10:41:26 (UTC+8)
2130tasks total=14270 running=3 done=3365 failed=0 blocked=0 pending=10902 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:404s;cuda2:trace_context:trace_bank__n1__Full-256-b__llama31_4b__pubmed__s42:201s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:1702s | GPU cuda1:7851/24564MiB:95%;cuda2:4595/24564MiB:97%;cuda3:10477/24564MiB:100%260813-10:38:24 (UTC+8)
2129tasks total=14270 running=3 done=3365 failed=0 blocked=0 pending=10902 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:223s;cuda2:trace_context:trace_bank__n1__Full-256-b__llama31_4b__pubmed__s42:20s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:1521s | GPU cuda1:7851/24564MiB:95%;cuda2:4595/24564MiB:97%;cuda3:10477/24564MiB:100%260813-10:35:22 (UTC+8)
2128tasks total=14270 running=3 done=3364 failed=0 blocked=0 pending=10903 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s3407:42s;cuda2:trace_context:trace_bank__n1__Full-256-a__llama31_4b__pubmed__s42:867s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:1340s | GPU cuda1:7851/24564MiB:100%;cuda2:4595/24564MiB:97%;cuda3:10477/24564MiB:100%260813-10:32:21 (UTC+8)
2127tasks total=14270 running=3 done=3363 failed=0 blocked=0 pending=10904 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:2195s;cuda2:trace_context:trace_bank__n1__Full-256-a__llama31_4b__pubmed__s42:686s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:1159s | GPU cuda1:7851/24564MiB:100%;cuda2:4595/24564MiB:97%;cuda3:10477/24564MiB:100%260813-10:29:21 (UTC+8)
2126tasks total=14270 running=3 done=3363 failed=0 blocked=0 pending=10904 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:2014s;cuda2:trace_context:trace_bank__n1__Full-256-a__llama31_4b__pubmed__s42:505s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:978s | GPU cuda1:7851/24564MiB:100%;cuda2:4595/24564MiB:100%;cuda3:10477/24564MiB:99%260813-10:26:20 (UTC+8)
2125tasks total=14270 running=3 done=3363 failed=0 blocked=0 pending=10904 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:1831s;cuda2:trace_context:trace_bank__n1__Full-256-a__llama31_4b__pubmed__s42:322s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:795s | GPU cuda1:7851/24564MiB:95%;cuda2:4595/24564MiB:93%;cuda3:10477/24564MiB:99%260813-10:23:19 (UTC+8)
2124tasks total=14270 running=3 done=3363 failed=0 blocked=0 pending=10904 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:1650s;cuda2:trace_context:trace_bank__n1__Full-256-a__llama31_4b__pubmed__s42:141s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:615s | GPU cuda1:7851/24564MiB:100%;cuda2:4595/24564MiB:97%;cuda3:10477/24564MiB:99%260813-10:20:16 (UTC+8)
2123tasks total=14270 running=3 done=3361 failed=0 blocked=0 pending=10906 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:1469s;cuda2:trace_context:trace_bank__n1__Full-128-b__llama31_4b__pubmed__s42:758s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:434s | GPU cuda1:7851/24564MiB:95%;cuda2:4575/24564MiB:97%;cuda3:10477/24564MiB:100%260813-10:17:15 (UTC+8)
2122tasks total=14270 running=3 done=3361 failed=0 blocked=0 pending=10906 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:1289s;cuda2:trace_context:trace_bank__n1__Full-128-b__llama31_4b__pubmed__s42:577s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:253s | GPU cuda1:7851/24564MiB:100%;cuda2:4575/24564MiB:97%;cuda3:10477/24564MiB:100%260813-10:14:14 (UTC+8)
2121tasks total=14270 running=3 done=3361 failed=0 blocked=0 pending=10906 | results done=1130/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:1105s;cuda2:trace_context:trace_bank__n1__Full-128-b__llama31_4b__pubmed__s42:394s;cuda3:sft:sft__trace__llama3_8b__pubmed__s42:70s | GPU cuda1:7851/24564MiB:96%;cuda2:4575/24564MiB:97%;cuda3:401/24564MiB:0%260813-10:11:13 (UTC+8)
2120tasks total=14270 running=3 done=3356 failed=0 blocked=0 pending=10911 | results done=1126/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:923s;cuda2:trace_context:trace_bank__n1__Full-128-b__llama31_4b__pubmed__s42:212s;cuda3:external_downstream:run__LinguGKD__llama31_4b__pubmed__s42__gcn3:50s | GPU cuda1:7851/24564MiB:92%;cuda2:4575/24564MiB:98%;cuda3:1949/24564MiB:0%260813-10:08:10 (UTC+8)
2119tasks total=14270 running=3 done=3352 failed=0 blocked=0 pending=10915 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:742s;cuda2:trace_context:trace_bank__n1__Full-128-b__llama31_4b__pubmed__s42:31s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:2225s | GPU cuda1:7851/24564MiB:100%;cuda2:4575/24564MiB:88%;cuda3:5485/24564MiB:100%260813-10:05:08 (UTC+8)
2118tasks total=14270 running=3 done=3351 failed=0 blocked=0 pending=10916 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:560s;cuda2:trace_context:trace_bank__n1__Full-128-a__llama31_4b__pubmed__s42:653s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:2043s | GPU cuda1:7851/24564MiB:100%;cuda2:4575/24564MiB:97%;cuda3:5485/24564MiB:0%260813-10:02:06 (UTC+8)
2117tasks total=14270 running=3 done=3351 failed=0 blocked=0 pending=10916 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:379s;cuda2:trace_context:trace_bank__n1__Full-128-a__llama31_4b__pubmed__s42:472s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:1862s | GPU cuda1:7851/24564MiB:100%;cuda2:4575/24564MiB:93%;cuda3:5485/24564MiB:90%260813-09:59:05 (UTC+8)
2116tasks total=14270 running=3 done=3351 failed=0 blocked=0 pending=10916 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:198s;cuda2:trace_context:trace_bank__n1__Full-128-a__llama31_4b__pubmed__s42:291s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:1681s | GPU cuda1:7851/24564MiB:95%;cuda2:4575/24564MiB:95%;cuda3:5485/24564MiB:100%260813-09:56:04 (UTC+8)
2115tasks total=14270 running=3 done=3351 failed=0 blocked=0 pending=10916 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s3407:17s;cuda2:trace_context:trace_bank__n1__Full-128-a__llama31_4b__pubmed__s42:110s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:1500s | GPU cuda1:7699/24564MiB:26%;cuda2:4575/24564MiB:95%;cuda3:5465/24564MiB:0%260813-09:53:03 (UTC+8)
2114tasks total=14270 running=3 done=3347 failed=0 blocked=0 pending=10920 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s3407:902s;cuda2:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__pubmed__s42:974s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:1317s | GPU cuda1:7851/24564MiB:97%;cuda2:4595/24564MiB:97%;cuda3:5465/24564MiB:100%260813-09:50:02 (UTC+8)
2113tasks total=14270 running=3 done=3347 failed=0 blocked=0 pending=10920 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s3407:721s;cuda2:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__pubmed__s42:793s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:1136s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:97%;cuda3:5465/24564MiB:0%260813-09:46:58 (UTC+8)
2112tasks total=14270 running=3 done=3347 failed=0 blocked=0 pending=10920 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s3407:540s;cuda2:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__pubmed__s42:612s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:955s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:97%;cuda3:5465/24564MiB:99%260813-09:43:57 (UTC+8)
2111tasks total=14270 running=3 done=3347 failed=0 blocked=0 pending=10920 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s3407:359s;cuda2:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__pubmed__s42:431s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:774s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:97%;cuda3:5465/24564MiB:95%260813-09:40:57 (UTC+8)
2110tasks total=14270 running=3 done=3347 failed=0 blocked=0 pending=10920 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s3407:177s;cuda2:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__pubmed__s42:250s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:592s | GPU cuda1:7851/24564MiB:95%;cuda2:4585/24564MiB:8%;cuda3:5425/24564MiB:96%260813-09:37:55 (UTC+8)
2109tasks total=14270 running=3 done=3346 failed=0 blocked=0 pending=10921 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s3407:1061s;cuda2:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__pubmed__s42:69s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:412s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:97%;cuda3:5425/24564MiB:96%260813-09:34:54 (UTC+8)
2108tasks total=14270 running=3 done=3345 failed=0 blocked=0 pending=10922 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s3407:880s;cuda2:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__pubmed__s42:935s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:230s | GPU cuda1:7851/24564MiB:95%;cuda2:4585/24564MiB:97%;cuda3:5425/24564MiB:100%260813-09:31:53 (UTC+8)
2107tasks total=14270 running=3 done=3345 failed=0 blocked=0 pending=10922 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s3407:699s;cuda2:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__pubmed__s42:754s;cuda3:lingu_bank:lingu_bank__llama31_4b__pubmed__s42:49s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:100%;cuda3:5385/24564MiB:100%260813-09:28:52 (UTC+8)
2106tasks total=14270 running=3 done=3344 failed=0 blocked=0 pending=10923 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s3407:516s;cuda2:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__pubmed__s42:572s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:8201s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:97%;cuda3:8469/24564MiB:100%260813-09:25:51 (UTC+8)
2105tasks total=14270 running=3 done=3344 failed=0 blocked=0 pending=10923 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s3407:335s;cuda2:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__pubmed__s42:391s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:8020s | GPU cuda1:7851/24564MiB:98%;cuda2:4585/24564MiB:97%;cuda3:8469/24564MiB:98%260813-09:22:48 (UTC+8)
2104tasks total=14270 running=3 done=3344 failed=0 blocked=0 pending=10923 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s3407:154s;cuda2:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__pubmed__s42:209s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:7838s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:97%;cuda3:8469/24564MiB:99%260813-09:19:47 (UTC+8)
2103tasks total=14270 running=3 done=3342 failed=0 blocked=0 pending=10925 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s3407:850s;cuda2:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__pubmed__s42:28s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:7657s | GPU cuda1:7851/24564MiB:98%;cuda2:4585/24564MiB:98%;cuda3:8469/24564MiB:100%260813-09:16:46 (UTC+8)
2102tasks total=14270 running=3 done=3340 failed=0 blocked=0 pending=10927 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s3407:669s;cuda2:trace_context:trace_bank__n1__Full-512-b__llama31_4b__pubmed__s42:1303s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:7477s | GPU cuda1:7851/24564MiB:97%;cuda2:4865/24564MiB:97%;cuda3:8469/24564MiB:100%260813-09:13:45 (UTC+8)
2101tasks total=14270 running=3 done=3340 failed=0 blocked=0 pending=10927 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s3407:488s;cuda2:trace_context:trace_bank__n1__Full-512-b__llama31_4b__pubmed__s42:1122s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:7296s | GPU cuda1:7851/24564MiB:98%;cuda2:4865/24564MiB:100%;cuda3:8469/24564MiB:100%260813-09:10:44 (UTC+8)
2100tasks total=14270 running=3 done=3340 failed=0 blocked=0 pending=10927 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s3407:307s;cuda2:trace_context:trace_bank__n1__Full-512-b__llama31_4b__pubmed__s42:941s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:7115s | GPU cuda1:7851/24564MiB:98%;cuda2:4865/24564MiB:85%;cuda3:8469/24564MiB:100%260813-09:07:43 (UTC+8)
2099tasks total=14270 running=3 done=3340 failed=0 blocked=0 pending=10927 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s3407:126s;cuda2:trace_context:trace_bank__n1__Full-512-b__llama31_4b__pubmed__s42:759s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:6933s | GPU cuda1:7851/24564MiB:97%;cuda2:4865/24564MiB:97%;cuda3:8469/24564MiB:99%260813-09:04:42 (UTC+8)
2098tasks total=14270 running=3 done=3339 failed=0 blocked=0 pending=10928 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s3407:820s;cuda2:trace_context:trace_bank__n1__Full-512-b__llama31_4b__pubmed__s42:579s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:6752s | GPU cuda1:7851/24564MiB:97%;cuda2:4865/24564MiB:97%;cuda3:8469/24564MiB:100%260813-09:01:41 (UTC+8)
2097tasks total=14270 running=3 done=3339 failed=0 blocked=0 pending=10928 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s3407:638s;cuda2:trace_context:trace_bank__n1__Full-512-b__llama31_4b__pubmed__s42:397s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:6571s | GPU cuda1:7851/24564MiB:98%;cuda2:4865/24564MiB:97%;cuda3:8469/24564MiB:100%260813-08:58:40 (UTC+8)
2096tasks total=14270 running=3 done=3339 failed=0 blocked=0 pending=10928 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s3407:457s;cuda2:trace_context:trace_bank__n1__Full-512-b__llama31_4b__pubmed__s42:216s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:6390s | GPU cuda1:7851/24564MiB:96%;cuda2:4865/24564MiB:96%;cuda3:8469/24564MiB:99%260813-08:55:39 (UTC+8)
2095tasks total=14270 running=3 done=3339 failed=0 blocked=0 pending=10928 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s3407:276s;cuda2:trace_context:trace_bank__n1__Full-512-b__llama31_4b__pubmed__s42:35s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:6209s | GPU cuda1:7851/24564MiB:95%;cuda2:4865/24564MiB:100%;cuda3:8469/24564MiB:99%260813-08:52:38 (UTC+8)
2094tasks total=14270 running=3 done=3338 failed=0 blocked=0 pending=10929 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s3407:95s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__pubmed__s42:1298s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:6028s | GPU cuda1:7851/24564MiB:95%;cuda2:4865/24564MiB:100%;cuda3:8469/24564MiB:99%260813-08:49:37 (UTC+8)
2093tasks total=14270 running=3 done=3336 failed=0 blocked=0 pending=10931 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s3407:1006s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__pubmed__s42:1118s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:5847s | GPU cuda1:7851/24564MiB:95%;cuda2:4865/24564MiB:97%;cuda3:8469/24564MiB:99%260813-08:46:35 (UTC+8)
2092tasks total=14270 running=3 done=3336 failed=0 blocked=0 pending=10931 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s3407:825s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__pubmed__s42:937s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:5666s | GPU cuda1:7851/24564MiB:97%;cuda2:4865/24564MiB:97%;cuda3:8469/24564MiB:94%260813-08:43:34 (UTC+8)
2091tasks total=14270 running=3 done=3336 failed=0 blocked=0 pending=10931 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s3407:644s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__pubmed__s42:755s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:5485s | GPU cuda1:7851/24564MiB:97%;cuda2:4865/24564MiB:100%;cuda3:8469/24564MiB:100%260813-08:40:34 (UTC+8)
2090tasks total=14270 running=3 done=3336 failed=0 blocked=0 pending=10931 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s3407:462s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__pubmed__s42:574s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:5303s | GPU cuda1:7851/24564MiB:97%;cuda2:4865/24564MiB:96%;cuda3:10373/24564MiB:100%260813-08:37:32 (UTC+8)
2089tasks total=14270 running=3 done=3336 failed=0 blocked=0 pending=10931 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s3407:281s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__pubmed__s42:393s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:5122s | GPU cuda1:7851/24564MiB:94%;cuda2:4865/24564MiB:96%;cuda3:10373/24564MiB:100%260813-08:34:31 (UTC+8)
2088tasks total=14270 running=3 done=3336 failed=0 blocked=0 pending=10931 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s3407:100s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__pubmed__s42:212s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:4941s | GPU cuda1:7851/24564MiB:97%;cuda2:4865/24564MiB:97%;cuda3:10373/24564MiB:100%260813-08:31:30 (UTC+8)
2087tasks total=14270 running=3 done=3335 failed=0 blocked=0 pending=10932 | results done=1124/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s3407:1010s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__pubmed__s42:31s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:4760s | GPU cuda1:7851/24564MiB:95%;cuda2:4865/24564MiB:96%;cuda3:10373/24564MiB:100%260813-08:28:29 (UTC+8)
2086tasks total=14270 running=3 done=3333 failed=0 blocked=0 pending=10934 | results done=1123/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s3407:829s;cuda2:g4_teacher:g4_teacher__llama31_4b__pubmed__s42:555s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:4579s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:98%;cuda3:10373/24564MiB:100%260813-08:25:28 (UTC+8)
2085tasks total=14270 running=3 done=3333 failed=0 blocked=0 pending=10934 | results done=1123/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s3407:647s;cuda2:g4_teacher:g4_teacher__llama31_4b__pubmed__s42:374s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:4398s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:97%;cuda3:10373/24564MiB:100%260813-08:22:27 (UTC+8)
2084tasks total=14270 running=3 done=3333 failed=0 blocked=0 pending=10934 | results done=1123/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s3407:466s;cuda2:g4_teacher:g4_teacher__llama31_4b__pubmed__s42:193s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:4217s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:82%;cuda3:10373/24564MiB:100%260813-08:19:26 (UTC+8)
2083tasks total=14270 running=3 done=3333 failed=0 blocked=0 pending=10934 | results done=1123/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s3407:286s;cuda2:g4_teacher:g4_teacher__llama31_4b__pubmed__s42:12s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:4036s | GPU cuda1:7851/24564MiB:94%;cuda2:4445/24564MiB:13%;cuda3:10373/24564MiB:100%260813-08:16:25 (UTC+8)
2082tasks total=14270 running=3 done=3325 failed=0 blocked=0 pending=10942 | results done=1116/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s3407:105s;cuda2:trace_downstream:run__O1_OrderA__llama31_4b__pubmed__s42__gcn3:55s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:3855s | GPU cuda1:7851/24564MiB:97%;cuda2:1095/24564MiB:73%;cuda3:10373/24564MiB:100%260813-08:13:24 (UTC+8)
2081tasks total=14270 running=3 done=3316 failed=0 blocked=0 pending=10951 | results done=1109/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s3407:1387s;cuda2:trace_downstream:run__C6_BudgetSweep__budget_256__llama31_4b__pubmed__s42__gcn3:7s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:3674s | GPU cuda1:7851/24564MiB:97%;cuda2:1095/24564MiB:58%;cuda3:10373/24564MiB:100%260813-08:10:23 (UTC+8)
2080tasks total=14270 running=3 done=3308 failed=0 blocked=0 pending=10959 | results done=1101/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s3407:1206s;cuda2:trace_downstream:run__C4_HopQuota__llama31_4b__pubmed__s42__gcn3:1s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:3493s | GPU cuda1:7851/24564MiB:97%;cuda2:401/24564MiB:1%;cuda3:10353/24564MiB:100%260813-08:07:22 (UTC+8)
2079tasks total=14270 running=3 done=3299 failed=0 blocked=0 pending=10968 | results done=1092/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s3407:1025s;cuda2:trace_downstream:run__A3_NoOrder__llama31_4b__pubmed__s42__gcn3:14s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:3312s | GPU cuda1:7851/24564MiB:96%;cuda2:1095/24564MiB:65%;cuda3:10353/24564MiB:100%260813-08:04:20 (UTC+8)
2078tasks total=14270 running=3 done=3289 failed=0 blocked=0 pending=10978 | results done=1082/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s3407:844s;cuda2:trace_downstream:run__T7_TRACEBilevelSoft__llama31_4b__pubmed__s42__sage3:17s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:3131s | GPU cuda1:7851/24564MiB:100%;cuda2:1077/24564MiB:58%;cuda3:10353/24564MiB:100%260813-08:01:20 (UTC+8)
2077tasks total=14270 running=3 done=3278 failed=0 blocked=0 pending=10989 | results done=1071/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s3407:663s;cuda2:trace_downstream:run__T6_TRACEProxy__llama31_4b__pubmed__s42__gcn1:10s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:2950s | GPU cuda1:7851/24564MiB:97%;cuda2:911/24564MiB:61%;cuda3:10353/24564MiB:100%260813-07:58:19 (UTC+8)
2076tasks total=14270 running=3 done=3271 failed=0 blocked=0 pending=10996 | results done=1064/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s3407:482s;cuda2:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__pubmed__s42__sage3:14s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:2769s | GPU cuda1:7851/24564MiB:0%;cuda2:1017/24564MiB:63%;cuda3:10353/24564MiB:100%260813-07:55:18 (UTC+8)
2075tasks total=14270 running=3 done=3268 failed=0 blocked=0 pending=10999 | results done=1061/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s3407:301s;cuda2:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__pubmed__s42__gcn3:45s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:2588s | GPU cuda1:7851/24564MiB:97%;cuda2:1095/24564MiB:66%;cuda3:10353/24564MiB:100%260813-07:52:16 (UTC+8)
2074tasks total=14270 running=3 done=3264 failed=0 blocked=0 pending=11003 | results done=1057/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s3407:120s;cuda2:trace_downstream:run__T3_TeacherOptGlobal__llama31_4b__pubmed__s42__sage2:6s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:2407s | GPU cuda1:7851/24564MiB:97%;cuda2:893/24564MiB:62%;cuda3:10353/24564MiB:100%260813-07:49:15 (UTC+8)
2073tasks total=14270 running=3 done=3255 failed=0 blocked=0 pending=11012 | results done=1053/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s3407:1400s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:2173s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:2226s | GPU cuda1:7851/24564MiB:100%;cuda2:5345/24564MiB:95%;cuda3:10353/24564MiB:100%260813-07:46:14 (UTC+8)
2072tasks total=14270 running=3 done=3255 failed=0 blocked=0 pending=11012 | results done=1053/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s3407:1220s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:1992s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:2045s | GPU cuda1:7851/24564MiB:97%;cuda2:5345/24564MiB:96%;cuda3:10353/24564MiB:100%260813-07:43:14 (UTC+8)
2071tasks total=14270 running=3 done=3255 failed=0 blocked=0 pending=11012 | results done=1053/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s3407:1038s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:1810s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:1864s | GPU cuda1:7851/24564MiB:97%;cuda2:5345/24564MiB:100%;cuda3:10353/24564MiB:100%260813-07:40:13 (UTC+8)
2070tasks total=14270 running=3 done=3255 failed=0 blocked=0 pending=11012 | results done=1053/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s3407:857s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:1630s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:1683s | GPU cuda1:7851/24564MiB:8%;cuda2:5345/24564MiB:96%;cuda3:10353/24564MiB:100%260813-07:37:11 (UTC+8)
2069tasks total=14270 running=3 done=3255 failed=0 blocked=0 pending=11012 | results done=1053/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s3407:676s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:1448s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:1502s | GPU cuda1:7851/24564MiB:98%;cuda2:5345/24564MiB:97%;cuda3:10353/24564MiB:100%260813-07:34:11 (UTC+8)
2068tasks total=14270 running=3 done=3255 failed=0 blocked=0 pending=11012 | results done=1053/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s3407:495s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:1268s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:1321s | GPU cuda1:7851/24564MiB:97%;cuda2:5345/24564MiB:100%;cuda3:10293/24564MiB:100%260813-07:31:09 (UTC+8)
2067tasks total=14270 running=3 done=3255 failed=0 blocked=0 pending=11012 | results done=1053/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s3407:314s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:1087s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:1140s | GPU cuda1:7851/24564MiB:100%;cuda2:5345/24564MiB:98%;cuda3:10293/24564MiB:99%260813-07:28:08 (UTC+8)
2066tasks total=14270 running=3 done=3255 failed=0 blocked=0 pending=11012 | results done=1053/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s3407:133s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:905s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:959s | GPU cuda1:7851/24564MiB:97%;cuda2:5345/24564MiB:96%;cuda3:10293/24564MiB:100%260813-07:25:07 (UTC+8)
2065tasks total=14270 running=3 done=3253 failed=0 blocked=0 pending=11014 | results done=1052/5050 | RUNNING cuda1:trace_analysis:trace_analysis__n1__llama3_8b__wikics__s3407:46s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:724s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:778s | GPU cuda1:1631/24564MiB:12%;cuda2:5345/24564MiB:100%;cuda3:10293/24564MiB:100%260813-07:22:06 (UTC+8)
2064tasks total=14270 running=3 done=3251 failed=0 blocked=0 pending=11016 | results done=1052/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:2038s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:543s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:596s | GPU cuda1:7851/24564MiB:97%;cuda2:5345/24564MiB:100%;cuda3:10293/24564MiB:99%260813-07:19:05 (UTC+8)
2063tasks total=14270 running=3 done=3251 failed=0 blocked=0 pending=11016 | results done=1052/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:1856s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:362s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:415s | GPU cuda1:7851/24564MiB:96%;cuda2:5325/24564MiB:96%;cuda3:10293/24564MiB:100%260813-07:16:04 (UTC+8)
2062tasks total=14270 running=3 done=3251 failed=0 blocked=0 pending=11016 | results done=1052/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:1675s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:181s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:234s | GPU cuda1:7851/24564MiB:100%;cuda2:5325/24564MiB:100%;cuda3:10193/24564MiB:100%260813-07:13:03 (UTC+8)
2061tasks total=14270 running=3 done=3251 failed=0 blocked=0 pending=11016 | results done=1052/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:1494s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__pubmed__s42:0s;cuda3:sft:sft__lingu__llama31_4b__pubmed__s42:53s | GPU cuda1:7851/24564MiB:96%;cuda2:14/24564MiB:0%;cuda3:2305/24564MiB:9%260813-07:10:02 (UTC+8)
2060tasks total=14270 running=3 done=3247 failed=0 blocked=0 pending=11020 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:1314s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:2010s;cuda3:trace_analysis:trace_analysis__n2__llama31_4b__wikics__s3407:23s | GPU cuda1:7851/24564MiB:100%;cuda2:5345/24564MiB:95%;cuda3:3535/24564MiB:99%260813-07:07:01 (UTC+8)
2059tasks total=14270 running=3 done=3245 failed=0 blocked=0 pending=11022 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:1133s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:1830s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:2025s | GPU cuda1:7851/24564MiB:100%;cuda2:5345/24564MiB:96%;cuda3:7349/24564MiB:100%260813-07:04:00 (UTC+8)
2058tasks total=14270 running=3 done=3245 failed=0 blocked=0 pending=11022 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:952s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:1649s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:1845s | GPU cuda1:7851/24564MiB:96%;cuda2:5345/24564MiB:96%;cuda3:7349/24564MiB:100%260813-07:00:59 (UTC+8)
2057tasks total=14270 running=3 done=3245 failed=0 blocked=0 pending=11022 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:771s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:1467s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:1663s | GPU cuda1:7851/24564MiB:96%;cuda2:5345/24564MiB:100%;cuda3:7349/24564MiB:100%260813-06:57:58 (UTC+8)
2056tasks total=14270 running=3 done=3245 failed=0 blocked=0 pending=11022 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:589s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:1286s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:1482s | GPU cuda1:7851/24564MiB:100%;cuda2:5345/24564MiB:100%;cuda3:7349/24564MiB:100%260813-06:54:57 (UTC+8)
2055tasks total=14270 running=3 done=3245 failed=0 blocked=0 pending=11022 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:408s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:1105s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:1301s | GPU cuda1:7851/24564MiB:95%;cuda2:5345/24564MiB:96%;cuda3:7349/24564MiB:100%260813-06:51:56 (UTC+8)
2054tasks total=14270 running=3 done=3245 failed=0 blocked=0 pending=11022 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:227s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:924s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:1120s | GPU cuda1:7851/24564MiB:94%;cuda2:5345/24564MiB:96%;cuda3:7349/24564MiB:96%260813-06:48:55 (UTC+8)
2053tasks total=14270 running=3 done=3245 failed=0 blocked=0 pending=11022 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s3407:46s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:743s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:939s | GPU cuda1:7851/24564MiB:96%;cuda2:5345/24564MiB:96%;cuda3:7349/24564MiB:100%260813-06:45:54 (UTC+8)
2052tasks total=14270 running=3 done=3244 failed=0 blocked=0 pending=11023 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:2035s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:562s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:758s | GPU cuda1:7851/24564MiB:96%;cuda2:5345/24564MiB:100%;cuda3:7349/24564MiB:100%260813-06:42:53 (UTC+8)
2051tasks total=14270 running=3 done=3244 failed=0 blocked=0 pending=11023 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:1854s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:381s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:576s | GPU cuda1:7851/24564MiB:96%;cuda2:5325/24564MiB:100%;cuda3:7349/24564MiB:100%260813-06:39:52 (UTC+8)
2050tasks total=14270 running=3 done=3244 failed=0 blocked=0 pending=11023 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:1673s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:200s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:396s | GPU cuda1:7851/24564MiB:100%;cuda2:5325/24564MiB:96%;cuda3:7349/24564MiB:100%260813-06:36:50 (UTC+8)
2049tasks total=14270 running=3 done=3244 failed=0 blocked=0 pending=11023 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:1492s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__pubmed__s42:18s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:214s | GPU cuda1:7851/24564MiB:100%;cuda2:5225/24564MiB:96%;cuda3:7349/24564MiB:100%260813-06:33:49 (UTC+8)
2048tasks total=14270 running=3 done=3241 failed=0 blocked=0 pending=11026 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:1311s;cuda2:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__pubmed__s42:1031s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s3407:33s | GPU cuda1:7851/24564MiB:96%;cuda2:4845/24564MiB:97%;cuda3:7349/24564MiB:96%260813-06:30:48 (UTC+8)
2047tasks total=14270 running=3 done=3240 failed=0 blocked=0 pending=11027 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:1129s;cuda2:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__pubmed__s42:850s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:2030s | GPU cuda1:7851/24564MiB:96%;cuda2:4845/24564MiB:97%;cuda3:7349/24564MiB:100%260813-06:27:47 (UTC+8)
2046tasks total=14270 running=3 done=3240 failed=0 blocked=0 pending=11027 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:948s;cuda2:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__pubmed__s42:669s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:1849s | GPU cuda1:7851/24564MiB:95%;cuda2:4845/24564MiB:97%;cuda3:7349/24564MiB:100%260813-06:24:46 (UTC+8)
2045tasks total=14270 running=3 done=3240 failed=0 blocked=0 pending=11027 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:767s;cuda2:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__pubmed__s42:488s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:1668s | GPU cuda1:7851/24564MiB:96%;cuda2:4845/24564MiB:97%;cuda3:7349/24564MiB:65%260813-06:21:45 (UTC+8)
2044tasks total=14270 running=3 done=3240 failed=0 blocked=0 pending=11027 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:587s;cuda2:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__pubmed__s42:307s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:1487s | GPU cuda1:7851/24564MiB:96%;cuda2:4845/24564MiB:97%;cuda3:7349/24564MiB:98%260813-06:18:44 (UTC+8)
2043tasks total=14270 running=3 done=3240 failed=0 blocked=0 pending=11027 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:405s;cuda2:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__pubmed__s42:126s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:1306s | GPU cuda1:7851/24564MiB:100%;cuda2:4825/24564MiB:97%;cuda3:7349/24564MiB:98%260813-06:15:43 (UTC+8)
2042tasks total=14270 running=3 done=3239 failed=0 blocked=0 pending=11028 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:224s;cuda2:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__pubmed__s42:1136s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:1125s | GPU cuda1:7851/24564MiB:96%;cuda2:4845/24564MiB:97%;cuda3:7349/24564MiB:100%260813-06:12:42 (UTC+8)
2041tasks total=14270 running=3 done=3239 failed=0 blocked=0 pending=11028 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s3407:43s;cuda2:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__pubmed__s42:955s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:944s | GPU cuda1:7851/24564MiB:96%;cuda2:4845/24564MiB:23%;cuda3:7349/24564MiB:100%260813-06:09:41 (UTC+8)
2040tasks total=14270 running=3 done=3237 failed=0 blocked=0 pending=11030 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s3407:1218s;cuda2:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__pubmed__s42:774s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:763s | GPU cuda1:7851/24564MiB:100%;cuda2:4845/24564MiB:95%;cuda3:7349/24564MiB:100%260813-06:06:40 (UTC+8)
2039tasks total=14270 running=3 done=3237 failed=0 blocked=0 pending=11030 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s3407:1037s;cuda2:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__pubmed__s42:593s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:582s | GPU cuda1:7851/24564MiB:97%;cuda2:4845/24564MiB:97%;cuda3:7349/24564MiB:100%260813-06:03:39 (UTC+8)
2038tasks total=14270 running=3 done=3237 failed=0 blocked=0 pending=11030 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s3407:856s;cuda2:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__pubmed__s42:412s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:401s | GPU cuda1:7851/24564MiB:97%;cuda2:4845/24564MiB:97%;cuda3:7349/24564MiB:99%260813-06:00:38 (UTC+8)
2037tasks total=14270 running=3 done=3237 failed=0 blocked=0 pending=11030 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s3407:675s;cuda2:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__pubmed__s42:231s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:220s | GPU cuda1:7851/24564MiB:97%;cuda2:4845/24564MiB:97%;cuda3:7349/24564MiB:98%260813-05:57:37 (UTC+8)
2036tasks total=14270 running=3 done=3237 failed=0 blocked=0 pending=11030 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s3407:494s;cuda2:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__pubmed__s42:49s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s3407:38s | GPU cuda1:7851/24564MiB:0%;cuda2:4785/24564MiB:0%;cuda3:7349/24564MiB:96%260813-05:54:36 (UTC+8)
2035tasks total=14270 running=3 done=3232 failed=0 blocked=0 pending=11035 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s3407:313s;cuda2:trace_context:trace_bank__base__HopQuota-256-b__llama31_4b__pubmed__s42:613s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s3407:1149s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:31%;cuda3:6789/24564MiB:98%260813-05:51:34 (UTC+8)
2034tasks total=14270 running=3 done=3232 failed=0 blocked=0 pending=11035 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s3407:132s;cuda2:trace_context:trace_bank__base__HopQuota-256-b__llama31_4b__pubmed__s42:432s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s3407:968s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:98%;cuda3:6789/24564MiB:99%260813-05:48:34 (UTC+8)
2033tasks total=14270 running=3 done=3231 failed=0 blocked=0 pending=11036 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s3407:1304s;cuda2:trace_context:trace_bank__base__HopQuota-256-b__llama31_4b__pubmed__s42:251s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s3407:787s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:95%;cuda3:6789/24564MiB:99%260813-05:45:33 (UTC+8)
2032tasks total=14270 running=3 done=3231 failed=0 blocked=0 pending=11036 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s3407:1123s;cuda2:trace_context:trace_bank__base__HopQuota-256-b__llama31_4b__pubmed__s42:70s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s3407:606s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:96%;cuda3:6789/24564MiB:98%260813-05:42:32 (UTC+8)
2031tasks total=14270 running=3 done=3230 failed=0 blocked=0 pending=11037 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s3407:942s;cuda2:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__pubmed__s42:630s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s3407:425s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:95%;cuda3:6789/24564MiB:100%260813-05:39:30 (UTC+8)
2030tasks total=14270 running=3 done=3230 failed=0 blocked=0 pending=11037 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s3407:761s;cuda2:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__pubmed__s42:449s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s3407:244s | GPU cuda1:7851/24564MiB:98%;cuda2:4565/24564MiB:97%;cuda3:6789/24564MiB:100%260813-05:36:30 (UTC+8)
2029tasks total=14270 running=3 done=3230 failed=0 blocked=0 pending=11037 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s3407:580s;cuda2:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__pubmed__s42:268s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s3407:63s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:95%;cuda3:6769/24564MiB:97%260813-05:33:29 (UTC+8)
2028tasks total=14270 running=3 done=3229 failed=0 blocked=0 pending=11038 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s3407:399s;cuda2:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__pubmed__s42:87s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s3407:1170s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:98%;cuda3:6789/24564MiB:53%260813-05:30:27 (UTC+8)
2027tasks total=14270 running=3 done=3226 failed=0 blocked=0 pending=11041 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s3407:217s;cuda2:trace_context:trace_bank__base__HopQuota-128-b__llama31_4b__pubmed__s42:637s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s3407:988s | GPU cuda1:7851/24564MiB:6%;cuda2:4565/24564MiB:95%;cuda3:6789/24564MiB:68%260813-05:27:26 (UTC+8)
2026tasks total=14270 running=3 done=3226 failed=0 blocked=0 pending=11041 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s3407:36s;cuda2:trace_context:trace_bank__base__HopQuota-128-b__llama31_4b__pubmed__s42:456s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s3407:807s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:97%;cuda3:6789/24564MiB:100%260813-05:24:25 (UTC+8)
2025tasks total=14270 running=3 done=3224 failed=0 blocked=0 pending=11043 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s3407:788s;cuda2:trace_context:trace_bank__base__HopQuota-128-b__llama31_4b__pubmed__s42:275s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s3407:626s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:98%;cuda3:6789/24564MiB:97%260813-05:21:24 (UTC+8)
2024tasks total=14270 running=3 done=3224 failed=0 blocked=0 pending=11043 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s3407:607s;cuda2:trace_context:trace_bank__base__HopQuota-128-b__llama31_4b__pubmed__s42:94s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s3407:445s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:95%;cuda3:6789/24564MiB:98%260813-05:18:23 (UTC+8)
2023tasks total=14270 running=3 done=3223 failed=0 blocked=0 pending=11044 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s3407:426s;cuda2:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__pubmed__s42:640s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s3407:264s | GPU cuda1:7851/24564MiB:98%;cuda2:4565/24564MiB:95%;cuda3:6789/24564MiB:100%260813-05:15:22 (UTC+8)
2022tasks total=14270 running=3 done=3223 failed=0 blocked=0 pending=11044 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s3407:245s;cuda2:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__pubmed__s42:459s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s3407:83s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:95%;cuda3:6769/24564MiB:97%260813-05:12:21 (UTC+8)
2021tasks total=14270 running=3 done=3221 failed=0 blocked=0 pending=11046 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s3407:64s;cuda2:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__pubmed__s42:277s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama31_4b__wikics__s3407:786s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:95%;cuda3:6529/24564MiB:99%260813-05:09:20 (UTC+8)
2020tasks total=14270 running=3 done=3220 failed=0 blocked=0 pending=11047 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s3407:812s;cuda2:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__pubmed__s42:96s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama31_4b__wikics__s3407:605s | GPU cuda1:7851/24564MiB:98%;cuda2:4565/24564MiB:95%;cuda3:6529/24564MiB:100%260813-05:06:19 (UTC+8)
2019tasks total=14270 running=3 done=3217 failed=0 blocked=0 pending=11050 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s3407:631s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:4429s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama31_4b__wikics__s3407:424s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:0%;cuda3:6509/24564MiB:60%260813-05:03:17 (UTC+8)
2018tasks total=14270 running=3 done=3217 failed=0 blocked=0 pending=11050 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s3407:449s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:4248s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama31_4b__wikics__s3407:242s | GPU cuda1:7851/24564MiB:98%;cuda2:5425/24564MiB:0%;cuda3:6509/24564MiB:99%260813-05:00:16 (UTC+8)
2017tasks total=14270 running=3 done=3217 failed=0 blocked=0 pending=11050 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s3407:268s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:4066s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama31_4b__wikics__s3407:61s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:0%;cuda3:6509/24564MiB:100%260813-04:57:15 (UTC+8)
2016tasks total=14270 running=3 done=3216 failed=0 blocked=0 pending=11051 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s3407:87s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:3885s;cuda3:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s3407:762s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:6529/24564MiB:98%260813-04:54:14 (UTC+8)
2015tasks total=14270 running=3 done=3214 failed=0 blocked=0 pending=11053 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s3407:707s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:3704s;cuda3:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s3407:581s | GPU cuda1:7851/24564MiB:98%;cuda2:5425/24564MiB:100%;cuda3:6529/24564MiB:98%260813-04:51:13 (UTC+8)
2014tasks total=14270 running=3 done=3214 failed=0 blocked=0 pending=11053 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s3407:526s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:3523s;cuda3:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s3407:400s | GPU cuda1:7851/24564MiB:95%;cuda2:5425/24564MiB:95%;cuda3:6509/24564MiB:98%260813-04:48:11 (UTC+8)
2013tasks total=14270 running=3 done=3214 failed=0 blocked=0 pending=11053 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s3407:345s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:3342s;cuda3:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s3407:219s | GPU cuda1:7851/24564MiB:96%;cuda2:5425/24564MiB:0%;cuda3:6509/24564MiB:97%260813-04:45:11 (UTC+8)
2012tasks total=14270 running=3 done=3214 failed=0 blocked=0 pending=11053 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s3407:163s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:3161s;cuda3:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s3407:38s | GPU cuda1:7851/24564MiB:96%;cuda2:5425/24564MiB:0%;cuda3:6509/24564MiB:98%260813-04:42:09 (UTC+8)
2011tasks total=14270 running=3 done=3211 failed=0 blocked=0 pending=11056 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s3407:781s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:2980s;cuda3:trace_context:trace_bank__n2__HopQuota-128-b__llama31_4b__wikics__s3407:531s | GPU cuda1:7851/24564MiB:95%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:98%260813-04:39:08 (UTC+8)
2010tasks total=14270 running=3 done=3211 failed=0 blocked=0 pending=11056 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s3407:600s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:2799s;cuda3:trace_context:trace_bank__n2__HopQuota-128-b__llama31_4b__wikics__s3407:350s | GPU cuda1:7851/24564MiB:95%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:99%260813-04:36:07 (UTC+8)
2009tasks total=14270 running=3 done=3211 failed=0 blocked=0 pending=11056 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s3407:419s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:2618s;cuda3:trace_context:trace_bank__n2__HopQuota-128-b__llama31_4b__wikics__s3407:169s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:98%260813-04:33:06 (UTC+8)
2008tasks total=14270 running=3 done=3210 failed=0 blocked=0 pending=11057 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s3407:238s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:2437s;cuda3:trace_context:trace_bank__n2__HopQuota-128-a__llama31_4b__wikics__s3407:660s | GPU cuda1:7851/24564MiB:95%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:98%260813-04:30:06 (UTC+8)
2007tasks total=14270 running=3 done=3210 failed=0 blocked=0 pending=11057 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s3407:57s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:2256s;cuda3:trace_context:trace_bank__n2__HopQuota-128-a__llama31_4b__wikics__s3407:478s | GPU cuda1:7851/24564MiB:96%;cuda2:5425/24564MiB:0%;cuda3:6469/24564MiB:99%260813-04:27:04 (UTC+8)
2006tasks total=14270 running=3 done=3208 failed=0 blocked=0 pending=11059 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:4622s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:2075s;cuda3:trace_context:trace_bank__n2__HopQuota-128-a__llama31_4b__wikics__s3407:298s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:98%260813-04:24:03 (UTC+8)
2005tasks total=14270 running=3 done=3208 failed=0 blocked=0 pending=11059 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:4441s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:1894s;cuda3:trace_context:trace_bank__n2__HopQuota-128-a__llama31_4b__wikics__s3407:117s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:98%260813-04:21:02 (UTC+8)
2004tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:4260s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:1713s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:4314s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:48%260813-04:18:01 (UTC+8)
2003tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:4079s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:1532s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:4133s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:46%260813-04:15:00 (UTC+8)
2002tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:3898s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:1351s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:3952s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260813-04:11:59 (UTC+8)
2001tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:3717s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:1170s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:3771s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260813-04:08:59 (UTC+8)
2000tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:3536s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:989s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:3590s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:46%260813-04:05:57 (UTC+8)
1999tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:3355s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:808s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:3410s | GPU cuda1:7851/24564MiB:96%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:71%260813-04:02:56 (UTC+8)
1998tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:3174s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:627s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:3228s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260813-03:59:56 (UTC+8)
1997tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:2993s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:446s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:3047s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260813-03:56:54 (UTC+8)
1996tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:2813s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:265s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:2867s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:46%260813-03:53:54 (UTC+8)
1995tasks total=14270 running=3 done=3206 failed=0 blocked=0 pending=11061 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:2631s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__pubmed__s42:84s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:2686s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:46%260813-03:50:53 (UTC+8)
1994tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:2451s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:4325s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:2505s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:48%260813-03:47:52 (UTC+8)
1993tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:2269s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:4144s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:2323s | GPU cuda1:7851/24564MiB:55%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:100%260813-03:44:51 (UTC+8)
1992tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:2089s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:3963s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:2143s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:100%260813-03:41:50 (UTC+8)
1991tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:1907s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:3781s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:1961s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260813-03:38:49 (UTC+8)
1990tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:1726s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:3601s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:1781s | GPU cuda1:7851/24564MiB:95%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:46%260813-03:35:47 (UTC+8)
1989tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:1546s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:3420s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:1600s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:46%260813-03:32:47 (UTC+8)
1988tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:1364s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:3238s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:1418s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:47%260813-03:29:46 (UTC+8)
1987tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:1183s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:3057s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:1237s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260813-03:26:44 (UTC+8)
1986tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:1002s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:2876s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:1056s | GPU cuda1:7851/24564MiB:96%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:100%260813-03:23:43 (UTC+8)
1985tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:821s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:2695s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:875s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:99%260813-03:20:42 (UTC+8)
1984tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:639s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:2514s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:694s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:7349/24564MiB:55%260813-03:17:41 (UTC+8)
1983tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:458s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:2332s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:512s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:0%;cuda3:7349/24564MiB:46%260813-03:14:40 (UTC+8)
1982tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:277s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:2151s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:331s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:7349/24564MiB:100%260813-03:11:38 (UTC+8)
1981tasks total=14270 running=3 done=3205 failed=0 blocked=0 pending=11062 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s3407:96s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:1970s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s3407:150s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7349/24564MiB:100%260813-03:08:37 (UTC+8)
1980tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:4686s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:1789s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:4462s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:96%260813-03:05:36 (UTC+8)
1979tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:4505s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:1608s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:4281s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:54%260813-03:02:35 (UTC+8)
1978tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:4324s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:1427s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:4100s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:46%260813-02:59:34 (UTC+8)
1977tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:4143s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:1246s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:3919s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:83%;cuda3:7369/24564MiB:100%260813-02:56:33 (UTC+8)
1976tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:3961s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:1065s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:3738s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:46%260813-02:53:32 (UTC+8)
1975tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:3781s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:884s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:3557s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:45%260813-02:50:31 (UTC+8)
1974tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:3599s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:703s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:3376s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260813-02:47:30 (UTC+8)
1973tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:3418s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:521s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:3194s | GPU cuda1:7851/24564MiB:41%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:48%260813-02:44:29 (UTC+8)
1972tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:3237s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:340s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:3013s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:0%;cuda3:7369/24564MiB:46%260813-02:41:27 (UTC+8)
1971tasks total=14270 running=3 done=3203 failed=0 blocked=0 pending=11064 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:3056s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__pubmed__s42:159s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:2832s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260813-02:38:26 (UTC+8)
1970tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:2875s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:3375s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:2651s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:7369/24564MiB:100%260813-02:35:25 (UTC+8)
1969tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:2694s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:3194s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:2470s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:7369/24564MiB:47%260813-02:32:24 (UTC+8)
1968tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:2513s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:3013s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:2289s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:0%;cuda3:7369/24564MiB:100%260813-02:29:23 (UTC+8)
1967tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:2332s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:2832s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:2108s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:97%;cuda3:7369/24564MiB:96%260813-02:26:22 (UTC+8)
1966tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:2151s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:2651s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:1927s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:97%;cuda3:7369/24564MiB:53%260813-02:23:21 (UTC+8)
1965tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:1970s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:2470s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:1746s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:7369/24564MiB:45%260813-02:20:20 (UTC+8)
1964tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:1789s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:2290s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:1566s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7369/24564MiB:96%260813-02:17:19 (UTC+8)
1963tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:1608s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:2108s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:1384s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:100%;cuda3:7369/24564MiB:13%260813-02:14:19 (UTC+8)
1962tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:1427s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:1927s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:1203s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:100%;cuda3:7369/24564MiB:46%260813-02:11:17 (UTC+8)
1961tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:1246s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:1746s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:1022s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:7369/24564MiB:96%260813-02:08:16 (UTC+8)
1960tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:1065s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:1565s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:841s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7349/24564MiB:100%260813-02:05:15 (UTC+8)
1959tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:883s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:1384s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:660s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:97%;cuda3:7349/24564MiB:52%260813-02:02:14 (UTC+8)
1958tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:702s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:1203s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:479s | GPU cuda1:7851/24564MiB:46%;cuda2:4885/24564MiB:0%;cuda3:7349/24564MiB:100%260813-01:59:13 (UTC+8)
1957tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:521s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:1022s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:298s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7349/24564MiB:100%260813-01:56:11 (UTC+8)
1956tasks total=14270 running=3 done=3200 failed=0 blocked=0 pending=11067 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:341s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:841s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s3407:117s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:7349/24564MiB:46%260813-01:53:11 (UTC+8)
1955tasks total=14270 running=3 done=3198 failed=0 blocked=0 pending=11069 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s3407:160s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:660s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:3360s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:100%260813-01:50:10 (UTC+8)
1954tasks total=14270 running=3 done=3196 failed=0 blocked=0 pending=11071 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:3917s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:479s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:3179s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:100%260813-01:47:09 (UTC+8)
1953tasks total=14270 running=3 done=3196 failed=0 blocked=0 pending=11071 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:3735s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:298s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:2998s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:100%;cuda3:6789/24564MiB:46%260813-01:44:08 (UTC+8)
1952tasks total=14270 running=3 done=3196 failed=0 blocked=0 pending=11071 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:3555s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__pubmed__s42:117s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:2817s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:46%260813-01:41:07 (UTC+8)
1951tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:3372s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:3285s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:2635s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:50%260813-01:38:06 (UTC+8)
1950tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:3191s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:3104s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:2453s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:99%260813-01:35:03 (UTC+8)
1949tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:3010s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:2923s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:2272s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:100%260813-01:32:02 (UTC+8)
1948tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:2829s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:2742s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:2091s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:46%260813-01:29:01 (UTC+8)
1947tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:2648s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:2561s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:1910s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:46%260813-01:26:00 (UTC+8)
1946tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:2466s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:2379s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:1728s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:55%260813-01:22:59 (UTC+8)
1945tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:2285s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:2198s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:1547s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:48%260813-01:19:57 (UTC+8)
1944tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:2104s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:2017s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:1367s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:6789/24564MiB:48%260813-01:16:56 (UTC+8)
1943tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:1922s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:1835s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:1185s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:13%;cuda3:6789/24564MiB:36%260813-01:13:55 (UTC+8)
1942tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:1741s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:1654s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:1004s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:48%260813-01:10:53 (UTC+8)
1941tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:1560s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:1473s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:822s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:41%260813-01:07:53 (UTC+8)
1940tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:1379s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:1292s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:641s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:100%260813-01:04:51 (UTC+8)
1939tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:1197s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:1110s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:460s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:100%260813-01:01:50 (UTC+8)
1938tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:1016s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:929s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:278s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:97%260813-00:58:49 (UTC+8)
1937tasks total=14270 running=3 done=3195 failed=0 blocked=0 pending=11072 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:835s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:748s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s3407:98s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:56%260813-00:55:47 (UTC+8)
1936tasks total=14270 running=3 done=3194 failed=0 blocked=0 pending=11073 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:654s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:567s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:3400s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:46%260813-00:52:46 (UTC+8)
1935tasks total=14270 running=3 done=3194 failed=0 blocked=0 pending=11073 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:473s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:386s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:3219s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:100%;cuda3:6789/24564MiB:46%260813-00:49:46 (UTC+8)
1934tasks total=14270 running=3 done=3194 failed=0 blocked=0 pending=11073 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:292s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:205s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:3038s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:97%260813-00:46:45 (UTC+8)
1933tasks total=14270 running=3 done=3194 failed=0 blocked=0 pending=11073 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s3407:111s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__pubmed__s42:24s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:2857s | GPU cuda1:7851/24564MiB:0%;cuda2:4549/24564MiB:0%;cuda3:6789/24564MiB:46%260813-00:43:43 (UTC+8)
1932tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:3791s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:2806s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:2676s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:46%260813-00:40:43 (UTC+8)
1931tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:3610s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:2625s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:2495s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:100%260813-00:37:42 (UTC+8)
1930tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:3429s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:2444s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:2314s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:97%260813-00:34:41 (UTC+8)
1929tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:3248s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:2263s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:2133s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:46%260813-00:31:40 (UTC+8)
1928tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:3068s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:2082s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:1952s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:48%260813-00:28:39 (UTC+8)
1927tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:2886s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:1900s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:1770s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:64%260813-00:25:38 (UTC+8)
1926tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:2705s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:1719s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:1590s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:97%260813-00:22:36 (UTC+8)
1925tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:2524s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:1539s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:1409s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:100%260813-00:19:35 (UTC+8)
1924tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:2342s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:1356s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:1227s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:13%260813-00:16:35 (UTC+8)
1923tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:2160s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:1174s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:1045s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:46%260813-00:13:33 (UTC+8)
1922tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:1979s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:994s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:864s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:49%260813-00:10:30 (UTC+8)
1921tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:1798s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:812s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:682s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:46%260813-00:07:30 (UTC+8)
1920tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:1617s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:631s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:502s | GPU cuda1:7851/24564MiB:99%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:100%260813-00:04:28 (UTC+8)
1919tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:1435s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:449s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:319s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:55%260813-00:01:27 (UTC+8)
1918tasks total=14270 running=3 done=3190 failed=0 blocked=0 pending=11077 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:1254s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:268s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s3407:139s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:6789/24564MiB:46%260812-23:58:25 (UTC+8)
1917tasks total=14270 running=3 done=3188 failed=0 blocked=0 pending=11079 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:1073s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__pubmed__s42:87s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:2824s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:95%;cuda3:6529/24564MiB:45%260812-23:55:24 (UTC+8)
1916tasks total=14270 running=3 done=3187 failed=0 blocked=0 pending=11080 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:892s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:2820s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:2643s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:6529/24564MiB:98%260812-23:52:24 (UTC+8)
1915tasks total=14270 running=3 done=3187 failed=0 blocked=0 pending=11080 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:711s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:2638s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:2462s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:98%;cuda3:6529/24564MiB:45%260812-23:49:23 (UTC+8)
1914tasks total=14270 running=3 done=3187 failed=0 blocked=0 pending=11080 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:528s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:2455s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:2279s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:0%;cuda3:6529/24564MiB:99%260812-23:46:22 (UTC+8)
1913tasks total=14270 running=3 done=3187 failed=0 blocked=0 pending=11080 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:347s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:2274s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:2098s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:0%;cuda3:6529/24564MiB:99%260812-23:43:18 (UTC+8)
1912tasks total=14270 running=3 done=3187 failed=0 blocked=0 pending=11080 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s3407:166s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:2093s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:1917s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:97%;cuda3:4625/24564MiB:0%260812-23:40:17 (UTC+8)
1911tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:3536s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:1912s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:1736s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:4625/24564MiB:0%260812-23:37:17 (UTC+8)
1910tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:3355s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:1731s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:1555s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:97%;cuda3:4625/24564MiB:0%260812-23:34:15 (UTC+8)
1909tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:3174s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:1550s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:1374s | GPU cuda1:7851/24564MiB:95%;cuda2:4585/24564MiB:0%;cuda3:4625/24564MiB:0%260812-23:31:14 (UTC+8)
1908tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:2992s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:1368s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:1191s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:4625/24564MiB:0%260812-23:28:14 (UTC+8)
1907tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:2810s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:1187s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:1010s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:4625/24564MiB:0%260812-23:25:11 (UTC+8)
1906tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:2630s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:1006s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:829s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:4625/24564MiB:0%260812-23:22:10 (UTC+8)
1905tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:2449s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:825s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:648s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:97%;cuda3:4625/24564MiB:97%260812-23:19:09 (UTC+8)
1904tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:2268s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:644s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:468s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:97%;cuda3:4625/24564MiB:0%260812-23:16:08 (UTC+8)
1903tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:2084s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:461s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:284s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:4625/24564MiB:97%260812-23:13:07 (UTC+8)
1902tasks total=14270 running=3 done=3185 failed=0 blocked=0 pending=11082 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:1903s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:280s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s3407:103s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:97%;cuda3:4625/24564MiB:97%260812-23:10:04 (UTC+8)
1901tasks total=14270 running=3 done=3184 failed=0 blocked=0 pending=11083 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:1720s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__pubmed__s42:97s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:2701s | GPU cuda1:7851/24564MiB:0%;cuda2:4585/24564MiB:0%;cuda3:4625/24564MiB:97%260812-23:07:03 (UTC+8)
1900tasks total=14270 running=3 done=3181 failed=0 blocked=0 pending=11086 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:1539s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:2627s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:2520s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:97%260812-23:04:00 (UTC+8)
1899tasks total=14270 running=3 done=3181 failed=0 blocked=0 pending=11086 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:1358s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:2446s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:2339s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:0%260812-23:00:59 (UTC+8)
1898tasks total=14270 running=3 done=3181 failed=0 blocked=0 pending=11086 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:1177s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:2265s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:2158s | GPU cuda1:7851/24564MiB:98%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:0%260812-22:57:57 (UTC+8)
1897tasks total=14270 running=3 done=3181 failed=0 blocked=0 pending=11086 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:994s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:2082s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:1975s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:95%;cuda3:4625/24564MiB:0%260812-22:54:56 (UTC+8)
1896tasks total=14270 running=3 done=3181 failed=0 blocked=0 pending=11086 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:813s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:1901s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:1794s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:0%260812-22:51:54 (UTC+8)
1895tasks total=14270 running=3 done=3181 failed=0 blocked=0 pending=11086 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:632s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:1720s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:1613s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:0%260812-22:48:53 (UTC+8)
1894tasks total=14270 running=3 done=3181 failed=0 blocked=0 pending=11086 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:451s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:1539s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:1432s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:4625/24564MiB:0%260812-22:45:52 (UTC+8)
1893tasks total=14270 running=3 done=3181 failed=0 blocked=0 pending=11086 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:270s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:1358s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:1251s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:0%260812-22:42:51 (UTC+8)
1892tasks total=14270 running=3 done=3181 failed=0 blocked=0 pending=11086 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s3407:88s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:1176s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:1069s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:98%;cuda3:4625/24564MiB:97%260812-22:39:50 (UTC+8)
1891tasks total=14270 running=3 done=3180 failed=0 blocked=0 pending=11087 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:3373s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:995s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:888s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:0%260812-22:36:47 (UTC+8)
1890tasks total=14270 running=3 done=3180 failed=0 blocked=0 pending=11087 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:3192s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:814s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:707s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:97%260812-22:33:47 (UTC+8)
1889tasks total=14270 running=3 done=3180 failed=0 blocked=0 pending=11087 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:3009s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:631s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:524s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:4625/24564MiB:0%260812-22:30:46 (UTC+8)
1888tasks total=14270 running=3 done=3180 failed=0 blocked=0 pending=11087 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:2827s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:449s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:343s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:97%260812-22:27:42 (UTC+8)
1887tasks total=14270 running=3 done=3180 failed=0 blocked=0 pending=11087 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:2647s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:268s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s3407:162s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:0%;cuda3:4625/24564MiB:0%260812-22:24:41 (UTC+8)
1886tasks total=14270 running=3 done=3178 failed=0 blocked=0 pending=11089 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:2466s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__pubmed__s42:88s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:2653s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4565/24564MiB:97%260812-22:21:40 (UTC+8)
1885tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:2285s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:2593s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:2472s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:95%;cuda3:4565/24564MiB:0%260812-22:18:39 (UTC+8)
1884tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:2104s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:2411s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:2291s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4565/24564MiB:0%260812-22:15:38 (UTC+8)
1883tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:1923s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:2230s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:2110s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4565/24564MiB:0%260812-22:12:37 (UTC+8)
1882tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:1742s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:2049s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:1929s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:98%;cuda3:4565/24564MiB:0%260812-22:09:36 (UTC+8)
1881tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:1560s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:1867s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:1747s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:4565/24564MiB:0%260812-22:06:35 (UTC+8)
1880tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:1378s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:1686s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:1566s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:4565/24564MiB:0%260812-22:03:33 (UTC+8)
1879tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:1195s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:1502s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:1382s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4565/24564MiB:0%260812-22:00:32 (UTC+8)
1878tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:1014s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:1322s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:1201s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:4565/24564MiB:0%260812-21:57:28 (UTC+8)
1877tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:833s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:1140s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:1020s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4565/24564MiB:0%260812-21:54:27 (UTC+8)
1876tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:651s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:959s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:839s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:4565/24564MiB:0%260812-21:51:26 (UTC+8)
1875tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:468s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:776s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:656s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:97%;cuda3:4565/24564MiB:0%260812-21:48:25 (UTC+8)
1874tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:287s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:595s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:474s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:65%;cuda3:6469/24564MiB:99%260812-21:45:22 (UTC+8)
1873tasks total=14270 running=3 done=3177 failed=0 blocked=0 pending=11090 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s3407:106s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:414s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:294s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:6469/24564MiB:99%260812-21:42:21 (UTC+8)
1872tasks total=14270 running=3 done=3175 failed=0 blocked=0 pending=11092 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:3248s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:233s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s3407:112s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:6469/24564MiB:46%260812-21:39:20 (UTC+8)
1871tasks total=14270 running=3 done=3174 failed=0 blocked=0 pending=11093 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:3067s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__pubmed__s42:52s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:2806s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:6469/24564MiB:43%260812-21:36:18 (UTC+8)
1870tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:2886s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:2409s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:2625s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:94%;cuda3:6469/24564MiB:5%260812-21:33:17 (UTC+8)
1869tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:2704s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:2227s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:2444s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:100%260812-21:30:16 (UTC+8)
1868tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:2523s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:2046s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:2263s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:48%260812-21:27:15 (UTC+8)
1867tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:2343s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:1865s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:2082s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:17%260812-21:24:14 (UTC+8)
1866tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:2162s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:1685s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:1901s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:96%;cuda3:6469/24564MiB:98%260812-21:21:13 (UTC+8)
1865tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:1980s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:1503s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:1720s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:46%260812-21:18:12 (UTC+8)
1864tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:1799s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:1322s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:1539s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:66%260812-21:15:11 (UTC+8)
1863tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:1618s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:1141s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:1358s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:46%260812-21:12:10 (UTC+8)
1862tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:1438s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:961s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:1177s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:46%260812-21:09:09 (UTC+8)
1861tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:1257s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:780s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:996s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:45%260812-21:06:08 (UTC+8)
1860tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:1075s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:598s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:814s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:99%260812-21:03:07 (UTC+8)
1859tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:893s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:416s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:633s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:67%260812-21:00:05 (UTC+8)
1858tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:713s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:235s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:452s | GPU cuda1:7851/24564MiB:95%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:5%260812-20:57:04 (UTC+8)
1857tasks total=14270 running=3 done=3171 failed=0 blocked=0 pending=11096 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:531s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__pubmed__s42:54s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:271s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:41%260812-20:54:03 (UTC+8)
1856tasks total=14270 running=3 done=3170 failed=0 blocked=0 pending=11097 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:350s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:2421s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s3407:89s | GPU cuda1:7851/24564MiB:49%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:46%260812-20:51:02 (UTC+8)
1855tasks total=14270 running=3 done=3168 failed=0 blocked=0 pending=11099 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s3407:169s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:2240s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:2235s | GPU cuda1:7851/24564MiB:98%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:98%260812-20:48:00 (UTC+8)
1854tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:3277s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:2059s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:2054s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260812-20:44:59 (UTC+8)
1853tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:3096s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:1878s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:1873s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:94%;cuda3:7369/24564MiB:100%260812-20:41:59 (UTC+8)
1852tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:2915s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:1697s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:1692s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260812-20:38:57 (UTC+8)
1851tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:2733s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:1515s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:1511s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:94%;cuda3:7369/24564MiB:100%260812-20:35:56 (UTC+8)
1850tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:2552s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:1334s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:1330s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:53%;cuda3:7369/24564MiB:100%260812-20:32:55 (UTC+8)
1849tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:2371s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:1153s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:1149s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-20:29:54 (UTC+8)
1848tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:2191s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:973s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:968s | GPU cuda1:7851/24564MiB:95%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260812-20:26:53 (UTC+8)
1847tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:2008s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:790s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:786s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-20:23:52 (UTC+8)
1846tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:1827s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:609s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:604s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:96%260812-20:20:50 (UTC+8)
1845tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:1646s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:428s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:424s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:94%;cuda3:7349/24564MiB:54%260812-20:17:49 (UTC+8)
1844tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:1465s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:247s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:243s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7349/24564MiB:100%260812-20:14:48 (UTC+8)
1843tasks total=14270 running=3 done=3167 failed=0 blocked=0 pending=11100 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:1284s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__pubmed__s42:67s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s3407:62s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:7349/24564MiB:100%260812-20:11:47 (UTC+8)
1842tasks total=14270 running=3 done=3163 failed=0 blocked=0 pending=11104 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:1103s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__pubmed__s42:1321s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:2211s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:100%;cuda3:7369/24564MiB:96%260812-20:08:46 (UTC+8)
1841tasks total=14270 running=3 done=3163 failed=0 blocked=0 pending=11104 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:922s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__pubmed__s42:1140s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:2031s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:96%;cuda3:7369/24564MiB:100%260812-20:05:45 (UTC+8)
1840tasks total=14270 running=3 done=3163 failed=0 blocked=0 pending=11104 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:741s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__pubmed__s42:959s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:1850s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:96%;cuda3:7369/24564MiB:100%260812-20:02:44 (UTC+8)
1839tasks total=14270 running=3 done=3163 failed=0 blocked=0 pending=11104 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:561s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__pubmed__s42:778s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:1669s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:97%;cuda3:7369/24564MiB:97%260812-19:59:43 (UTC+8)
1838tasks total=14270 running=3 done=3163 failed=0 blocked=0 pending=11104 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:380s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__pubmed__s42:598s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:1488s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:97%;cuda3:7369/24564MiB:100%260812-19:56:42 (UTC+8)
1837tasks total=14270 running=3 done=3163 failed=0 blocked=0 pending=11104 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:199s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__pubmed__s42:416s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:1307s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:100%;cuda3:7369/24564MiB:100%260812-19:53:41 (UTC+8)
1836tasks total=14270 running=3 done=3163 failed=0 blocked=0 pending=11104 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s3407:16s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__pubmed__s42:234s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:1125s | GPU cuda1:7695/24564MiB:3%;cuda2:4885/24564MiB:96%;cuda3:7369/24564MiB:96%260812-19:50:40 (UTC+8)
1835tasks total=14270 running=3 done=3161 failed=0 blocked=0 pending=11106 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:2191s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__pubmed__s42:53s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:943s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7369/24564MiB:100%260812-19:47:38 (UTC+8)
1834tasks total=14270 running=3 done=3160 failed=0 blocked=0 pending=11107 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:2010s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__pubmed__s42:1321s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:763s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7369/24564MiB:100%260812-19:44:37 (UTC+8)
1833tasks total=14270 running=3 done=3160 failed=0 blocked=0 pending=11107 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:1829s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__pubmed__s42:1140s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:581s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:96%;cuda3:7369/24564MiB:98%260812-19:41:36 (UTC+8)
1832tasks total=14270 running=3 done=3160 failed=0 blocked=0 pending=11107 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:1648s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__pubmed__s42:958s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:400s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:97%;cuda3:7349/24564MiB:100%260812-19:38:35 (UTC+8)
1831tasks total=14270 running=3 done=3160 failed=0 blocked=0 pending=11107 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:1467s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__pubmed__s42:778s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:219s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:100%;cuda3:7349/24564MiB:100%260812-19:35:34 (UTC+8)
1830tasks total=14270 running=3 done=3160 failed=0 blocked=0 pending=11107 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:1286s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__pubmed__s42:596s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s3407:38s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7349/24564MiB:100%260812-19:32:33 (UTC+8)
1829tasks total=14270 running=3 done=3158 failed=0 blocked=0 pending=11109 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:1104s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__pubmed__s42:414s;cuda3:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s3407:1209s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:6809/24564MiB:97%260812-19:29:31 (UTC+8)
1828tasks total=14270 running=3 done=3158 failed=0 blocked=0 pending=11109 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:923s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__pubmed__s42:233s;cuda3:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s3407:1028s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:100%;cuda3:6809/24564MiB:100%260812-19:26:29 (UTC+8)
1827tasks total=14270 running=3 done=3158 failed=0 blocked=0 pending=11109 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:742s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__pubmed__s42:53s;cuda3:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s3407:848s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:100%;cuda3:6789/24564MiB:100%260812-19:23:29 (UTC+8)
1826tasks total=14270 running=3 done=3155 failed=0 blocked=0 pending=11112 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:561s;cuda2:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__pubmed__s42:683s;cuda3:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s3407:667s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:97%;cuda3:6789/24564MiB:97%260812-19:20:28 (UTC+8)
1825tasks total=14270 running=3 done=3155 failed=0 blocked=0 pending=11112 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:380s;cuda2:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__pubmed__s42:501s;cuda3:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s3407:485s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:95%;cuda3:6789/24564MiB:97%260812-19:17:27 (UTC+8)
1824tasks total=14270 running=3 done=3155 failed=0 blocked=0 pending=11112 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:198s;cuda2:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__pubmed__s42:319s;cuda3:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s3407:303s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:95%;cuda3:6789/24564MiB:100%260812-19:14:26 (UTC+8)
1823tasks total=14270 running=3 done=3155 failed=0 blocked=0 pending=11112 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s3407:17s;cuda2:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__pubmed__s42:138s;cuda3:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s3407:122s | GPU cuda1:7695/24564MiB:25%;cuda2:4565/24564MiB:97%;cuda3:6789/24564MiB:97%260812-19:11:24 (UTC+8)
1822tasks total=14270 running=3 done=3152 failed=0 blocked=0 pending=11115 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:2192s;cuda2:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__pubmed__s42:768s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s3407:1290s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:6809/24564MiB:97%260812-19:08:22 (UTC+8)
1821tasks total=14270 running=3 done=3152 failed=0 blocked=0 pending=11115 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:2011s;cuda2:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__pubmed__s42:588s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s3407:1109s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:95%;cuda3:6809/24564MiB:98%260812-19:05:22 (UTC+8)
1820tasks total=14270 running=3 done=3152 failed=0 blocked=0 pending=11115 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:1830s;cuda2:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__pubmed__s42:406s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s3407:928s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:97%;cuda3:6789/24564MiB:100%260812-19:02:21 (UTC+8)
1819tasks total=14270 running=3 done=3152 failed=0 blocked=0 pending=11115 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:1649s;cuda2:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__pubmed__s42:225s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s3407:747s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:95%;cuda3:6789/24564MiB:99%260812-18:59:19 (UTC+8)
1818tasks total=14270 running=3 done=3152 failed=0 blocked=0 pending=11115 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:1468s;cuda2:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__pubmed__s42:44s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s3407:566s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:93%;cuda3:6789/24564MiB:98%260812-18:56:18 (UTC+8)
1817tasks total=14270 running=3 done=3149 failed=0 blocked=0 pending=11118 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:1287s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:2437s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s3407:385s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:6789/24564MiB:100%260812-18:53:18 (UTC+8)
1816tasks total=14270 running=3 done=3149 failed=0 blocked=0 pending=11118 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:1106s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:2256s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s3407:204s | GPU cuda1:7851/24564MiB:96%;cuda2:5425/24564MiB:100%;cuda3:6789/24564MiB:100%260812-18:50:17 (UTC+8)
1815tasks total=14270 running=3 done=3149 failed=0 blocked=0 pending=11118 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:924s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:2074s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s3407:22s | GPU cuda1:7851/24564MiB:96%;cuda2:5425/24564MiB:95%;cuda3:6789/24564MiB:100%260812-18:47:16 (UTC+8)
1814tasks total=14270 running=3 done=3147 failed=0 blocked=0 pending=11120 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:743s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:1893s;cuda3:trace_context:trace_bank__n2__Semantic-128-b__llama31_4b__wikics__s3407:606s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:100%260812-18:44:13 (UTC+8)
1813tasks total=14270 running=3 done=3147 failed=0 blocked=0 pending=11120 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:372s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:1522s;cuda3:trace_context:trace_bank__n2__Semantic-128-b__llama31_4b__wikics__s3407:234s | GPU cuda1:7851/24564MiB:95%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:100%260812-18:38:02 (UTC+8)
1812tasks total=14270 running=3 done=3147 failed=0 blocked=0 pending=11120 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:188s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:1338s;cuda3:trace_context:trace_bank__n2__Semantic-128-b__llama31_4b__wikics__s3407:50s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:99%;cuda3:6469/24564MiB:46%260812-18:35:01 (UTC+8)
1811tasks total=14270 running=3 done=3146 failed=0 blocked=0 pending=11121 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s3407:7s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:1157s;cuda3:trace_context:trace_bank__n2__Semantic-128-a__llama31_4b__wikics__s3407:632s | GPU cuda1:7563/24564MiB:16%;cuda2:5425/24564MiB:96%;cuda3:6469/24564MiB:98%260812-18:31:57 (UTC+8)
1810tasks total=14270 running=3 done=3144 failed=0 blocked=0 pending=11123 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s3407:1314s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:976s;cuda3:trace_context:trace_bank__n2__Semantic-128-a__llama31_4b__wikics__s3407:450s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:98%260812-18:28:56 (UTC+8)
1809tasks total=14270 running=3 done=3144 failed=0 blocked=0 pending=11123 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s3407:1133s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:795s;cuda3:trace_context:trace_bank__n2__Semantic-128-a__llama31_4b__wikics__s3407:269s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:99%260812-18:25:55 (UTC+8)
1808tasks total=14270 running=3 done=3144 failed=0 blocked=0 pending=11123 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s3407:952s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:614s;cuda3:trace_context:trace_bank__n2__Semantic-128-a__llama31_4b__wikics__s3407:88s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:98%260812-18:22:54 (UTC+8)
1807tasks total=14270 running=3 done=3142 failed=0 blocked=0 pending=11125 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s3407:770s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:432s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:2218s | GPU cuda1:7851/24564MiB:99%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260812-18:19:53 (UTC+8)
1806tasks total=14270 running=3 done=3142 failed=0 blocked=0 pending=11125 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s3407:585s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:247s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:2033s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-18:16:50 (UTC+8)
1805tasks total=14270 running=3 done=3142 failed=0 blocked=0 pending=11125 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s3407:404s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__pubmed__s42:66s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:1852s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-18:13:46 (UTC+8)
1804tasks total=14270 running=3 done=3141 failed=0 blocked=0 pending=11126 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s3407:223s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:2450s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:1671s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260812-18:10:45 (UTC+8)
1803tasks total=14270 running=3 done=3141 failed=0 blocked=0 pending=11126 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s3407:42s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:2269s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:1490s | GPU cuda1:7851/24564MiB:13%;cuda2:5425/24564MiB:95%;cuda3:7349/24564MiB:97%260812-18:07:44 (UTC+8)
1802tasks total=14270 running=3 done=3140 failed=0 blocked=0 pending=11127 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s3407:1349s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:2088s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:1309s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:94%;cuda3:7349/24564MiB:100%260812-18:04:43 (UTC+8)
1801tasks total=14270 running=3 done=3140 failed=0 blocked=0 pending=11127 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s3407:1168s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:1907s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:1128s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:95%;cuda3:7349/24564MiB:100%260812-18:01:42 (UTC+8)
1800tasks total=14270 running=3 done=3140 failed=0 blocked=0 pending=11127 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s3407:985s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:1725s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:946s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:95%;cuda3:7349/24564MiB:100%260812-17:58:41 (UTC+8)
1799tasks total=14270 running=3 done=3140 failed=0 blocked=0 pending=11127 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s3407:805s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:1544s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:765s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:7349/24564MiB:100%260812-17:55:39 (UTC+8)
1798tasks total=14270 running=3 done=3140 failed=0 blocked=0 pending=11127 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s3407:622s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:1362s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:582s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:96%;cuda3:7349/24564MiB:100%260812-17:52:38 (UTC+8)
1797tasks total=14270 running=3 done=3140 failed=0 blocked=0 pending=11127 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s3407:441s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:1180s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:401s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:7349/24564MiB:100%260812-17:49:36 (UTC+8)
1796tasks total=14270 running=3 done=3140 failed=0 blocked=0 pending=11127 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s3407:260s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:1000s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:220s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7349/24564MiB:66%260812-17:46:34 (UTC+8)
1795tasks total=14270 running=3 done=3140 failed=0 blocked=0 pending=11127 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s3407:79s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:819s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s3407:39s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7349/24564MiB:100%260812-17:43:34 (UTC+8)
1794tasks total=14270 running=3 done=3137 failed=0 blocked=0 pending=11130 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s3407:804s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:638s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:2163s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-17:40:33 (UTC+8)
1793tasks total=14270 running=3 done=3137 failed=0 blocked=0 pending=11130 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s3407:622s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:456s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:1982s | GPU cuda1:7851/24564MiB:96%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:98%260812-17:37:32 (UTC+8)
1792tasks total=14270 running=3 done=3137 failed=0 blocked=0 pending=11130 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s3407:441s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:275s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:1801s | GPU cuda1:7851/24564MiB:58%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:96%260812-17:34:30 (UTC+8)
1791tasks total=14270 running=3 done=3137 failed=0 blocked=0 pending=11130 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s3407:260s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__pubmed__s42:94s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:1620s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-17:31:29 (UTC+8)
1790tasks total=14270 running=3 done=3134 failed=0 blocked=0 pending=11133 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s3407:79s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__pubmed__s42:1340s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:1439s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:80%;cuda3:7349/24564MiB:100%260812-17:28:28 (UTC+8)
1789tasks total=14270 running=3 done=3133 failed=0 blocked=0 pending=11134 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-a__llama3_8b__wikics__s3407:801s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__pubmed__s42:1159s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:1258s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:75%;cuda3:7349/24564MiB:100%260812-17:25:27 (UTC+8)
1788tasks total=14270 running=3 done=3133 failed=0 blocked=0 pending=11134 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-a__llama3_8b__wikics__s3407:620s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__pubmed__s42:978s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:1077s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:97%;cuda3:7349/24564MiB:100%260812-17:22:26 (UTC+8)
1787tasks total=14270 running=3 done=3133 failed=0 blocked=0 pending=11134 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-a__llama3_8b__wikics__s3407:439s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__pubmed__s42:797s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:896s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:100%;cuda3:7349/24564MiB:96%260812-17:19:25 (UTC+8)
1786tasks total=14270 running=3 done=3133 failed=0 blocked=0 pending=11134 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-a__llama3_8b__wikics__s3407:258s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__pubmed__s42:616s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:715s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:52%;cuda3:7349/24564MiB:100%260812-17:16:24 (UTC+8)
1785tasks total=14270 running=3 done=3133 failed=0 blocked=0 pending=11134 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-a__llama3_8b__wikics__s3407:77s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__pubmed__s42:435s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:534s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:96%;cuda3:7349/24564MiB:100%260812-17:13:23 (UTC+8)
1784tasks total=14270 running=3 done=3131 failed=0 blocked=0 pending=11136 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:2228s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__pubmed__s42:254s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:353s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:96%;cuda3:7349/24564MiB:100%260812-17:10:22 (UTC+8)
1783tasks total=14270 running=3 done=3131 failed=0 blocked=0 pending=11136 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:2047s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__pubmed__s42:73s;cuda3:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s3407:172s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:100%;cuda3:7349/24564MiB:100%260812-17:07:21 (UTC+8)
1782tasks total=14270 running=3 done=3128 failed=0 blocked=0 pending=11139 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:1866s;cuda2:trace_context:trace_bank__base__Random-512-a__llama31_4b__pubmed__s42:1327s;cuda3:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s3407:1327s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:6789/24564MiB:100%260812-17:04:20 (UTC+8)
1781tasks total=14270 running=3 done=3128 failed=0 blocked=0 pending=11139 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:1685s;cuda2:trace_context:trace_bank__base__Random-512-a__llama31_4b__pubmed__s42:1146s;cuda3:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s3407:1146s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:100%260812-17:01:19 (UTC+8)
1780tasks total=14270 running=3 done=3128 failed=0 blocked=0 pending=11139 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:1504s;cuda2:trace_context:trace_bank__base__Random-512-a__llama31_4b__pubmed__s42:965s;cuda3:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s3407:965s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:95%260812-16:58:18 (UTC+8)
1779tasks total=14270 running=3 done=3128 failed=0 blocked=0 pending=11139 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:1323s;cuda2:trace_context:trace_bank__base__Random-512-a__llama31_4b__pubmed__s42:784s;cuda3:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s3407:784s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:99%;cuda3:6789/24564MiB:100%260812-16:55:17 (UTC+8)
1778tasks total=14270 running=3 done=3128 failed=0 blocked=0 pending=11139 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:1142s;cuda2:trace_context:trace_bank__base__Random-512-a__llama31_4b__pubmed__s42:603s;cuda3:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s3407:603s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:6789/24564MiB:99%260812-16:52:16 (UTC+8)
1777tasks total=14270 running=3 done=3128 failed=0 blocked=0 pending=11139 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:961s;cuda2:trace_context:trace_bank__base__Random-512-a__llama31_4b__pubmed__s42:422s;cuda3:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s3407:422s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:6789/24564MiB:99%260812-16:49:15 (UTC+8)
1776tasks total=14270 running=3 done=3128 failed=0 blocked=0 pending=11139 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:780s;cuda2:trace_context:trace_bank__base__Random-512-a__llama31_4b__pubmed__s42:241s;cuda3:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s3407:241s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:0%;cuda3:6789/24564MiB:100%260812-16:46:15 (UTC+8)
1775tasks total=14270 running=3 done=3128 failed=0 blocked=0 pending=11139 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:599s;cuda2:trace_context:trace_bank__base__Random-512-a__llama31_4b__pubmed__s42:60s;cuda3:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s3407:60s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:97%260812-16:43:14 (UTC+8)
1774tasks total=14270 running=3 done=3124 failed=0 blocked=0 pending=11143 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:417s;cuda2:trace_context:trace_bank__base__Random-256-b__llama31_4b__pubmed__s42:878s;cuda3:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s3407:1211s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:100%260812-16:40:12 (UTC+8)
1773tasks total=14270 running=3 done=3124 failed=0 blocked=0 pending=11143 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:235s;cuda2:trace_context:trace_bank__base__Random-256-b__llama31_4b__pubmed__s42:696s;cuda3:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s3407:1029s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:100%260812-16:37:11 (UTC+8)
1772tasks total=14270 running=3 done=3124 failed=0 blocked=0 pending=11143 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s3407:54s;cuda2:trace_context:trace_bank__base__Random-256-b__llama31_4b__pubmed__s42:515s;cuda3:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s3407:848s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:100%260812-16:34:09 (UTC+8)
1771tasks total=14270 running=3 done=3123 failed=0 blocked=0 pending=11144 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:2200s;cuda2:trace_context:trace_bank__base__Random-256-b__llama31_4b__pubmed__s42:333s;cuda3:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s3407:666s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:96%;cuda3:6789/24564MiB:98%260812-16:31:08 (UTC+8)
1770tasks total=14270 running=3 done=3123 failed=0 blocked=0 pending=11144 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:2018s;cuda2:trace_context:trace_bank__base__Random-256-b__llama31_4b__pubmed__s42:151s;cuda3:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s3407:484s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:100%260812-16:28:06 (UTC+8)
1769tasks total=14270 running=3 done=3122 failed=0 blocked=0 pending=11145 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:1838s;cuda2:trace_context:trace_bank__base__Random-256-a__llama31_4b__pubmed__s42:971s;cuda3:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s3407:303s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:100%260812-16:25:04 (UTC+8)
1768tasks total=14270 running=3 done=3122 failed=0 blocked=0 pending=11145 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:1657s;cuda2:trace_context:trace_bank__base__Random-256-a__llama31_4b__pubmed__s42:790s;cuda3:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s3407:122s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:97%;cuda3:6789/24564MiB:99%260812-16:22:03 (UTC+8)
1767tasks total=14270 running=3 done=3120 failed=0 blocked=0 pending=11147 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:1474s;cuda2:trace_context:trace_bank__base__Random-256-a__llama31_4b__pubmed__s42:608s;cuda3:trace_context:trace_bank__n2__Random-256-b__llama31_4b__wikics__s3407:881s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-16:19:02 (UTC+8)
1766tasks total=14270 running=3 done=3120 failed=0 blocked=0 pending=11147 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:1290s;cuda2:trace_context:trace_bank__base__Random-256-a__llama31_4b__pubmed__s42:424s;cuda3:trace_context:trace_bank__n2__Random-256-b__llama31_4b__wikics__s3407:697s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-16:16:00 (UTC+8)
1765tasks total=14270 running=3 done=3120 failed=0 blocked=0 pending=11147 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:1110s;cuda2:trace_context:trace_bank__base__Random-256-a__llama31_4b__pubmed__s42:243s;cuda3:trace_context:trace_bank__n2__Random-256-b__llama31_4b__wikics__s3407:516s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-16:12:56 (UTC+8)
1764tasks total=14270 running=3 done=3120 failed=0 blocked=0 pending=11147 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:929s;cuda2:trace_context:trace_bank__base__Random-256-a__llama31_4b__pubmed__s42:62s;cuda3:trace_context:trace_bank__n2__Random-256-b__llama31_4b__wikics__s3407:336s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-16:09:55 (UTC+8)
1763tasks total=14270 running=3 done=3117 failed=0 blocked=0 pending=11150 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:748s;cuda2:trace_context:trace_bank__base__Random-128-b__llama31_4b__pubmed__s42:672s;cuda3:trace_context:trace_bank__n2__Random-256-b__llama31_4b__wikics__s3407:155s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:95%;cuda3:6529/24564MiB:99%260812-16:06:54 (UTC+8)
1762tasks total=14270 running=3 done=3116 failed=0 blocked=0 pending=11151 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:567s;cuda2:trace_context:trace_bank__base__Random-128-b__llama31_4b__pubmed__s42:491s;cuda3:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s3407:910s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:98%;cuda3:6529/24564MiB:65%260812-16:03:53 (UTC+8)
1761tasks total=14270 running=3 done=3116 failed=0 blocked=0 pending=11151 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:386s;cuda2:trace_context:trace_bank__base__Random-128-b__llama31_4b__pubmed__s42:310s;cuda3:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s3407:729s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:65%;cuda3:6529/24564MiB:97%260812-16:00:52 (UTC+8)
1760tasks total=14270 running=3 done=3116 failed=0 blocked=0 pending=11151 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:205s;cuda2:trace_context:trace_bank__base__Random-128-b__llama31_4b__pubmed__s42:129s;cuda3:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s3407:548s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:97%;cuda3:6529/24564MiB:100%260812-15:57:51 (UTC+8)
1759tasks total=14270 running=3 done=3115 failed=0 blocked=0 pending=11152 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s3407:24s;cuda2:trace_context:trace_bank__base__Random-128-a__llama31_4b__pubmed__s42:732s;cuda3:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s3407:367s | GPU cuda1:7835/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:6529/24564MiB:98%260812-15:54:50 (UTC+8)
1758tasks total=14270 running=3 done=3113 failed=0 blocked=0 pending=11154 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s3407:1306s;cuda2:trace_context:trace_bank__base__Random-128-a__llama31_4b__pubmed__s42:551s;cuda3:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s3407:186s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:97%;cuda3:6529/24564MiB:99%260812-15:51:50 (UTC+8)
1757tasks total=14270 running=3 done=3113 failed=0 blocked=0 pending=11154 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s3407:1123s;cuda2:trace_context:trace_bank__base__Random-128-a__llama31_4b__pubmed__s42:369s;cuda3:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s3407:4s | GPU cuda1:7851/24564MiB:61%;cuda2:4565/24564MiB:98%;cuda3:2305/24564MiB:49%260812-15:48:49 (UTC+8)
1756tasks total=14270 running=3 done=3111 failed=0 blocked=0 pending=11156 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s3407:941s;cuda2:trace_context:trace_bank__base__Random-128-a__llama31_4b__pubmed__s42:186s;cuda3:trace_context:trace_bank__n2__Random-128-b__llama31_4b__wikics__s3407:561s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:95%;cuda3:6469/24564MiB:98%260812-15:45:46 (UTC+8)
1755tasks total=14270 running=3 done=3111 failed=0 blocked=0 pending=11156 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s3407:760s;cuda2:trace_context:trace_bank__base__Random-128-a__llama31_4b__pubmed__s42:6s;cuda3:trace_context:trace_bank__n2__Random-128-b__llama31_4b__wikics__s3407:380s | GPU cuda1:7851/24564MiB:97%;cuda2:401/24564MiB:0%;cuda3:6469/24564MiB:100%260812-15:42:44 (UTC+8)
1754tasks total=14270 running=3 done=3108 failed=0 blocked=0 pending=11159 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s3407:578s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:2398s;cuda3:trace_context:trace_bank__n2__Random-128-b__llama31_4b__wikics__s3407:198s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:98%260812-15:39:43 (UTC+8)
1753tasks total=14270 running=3 done=3108 failed=0 blocked=0 pending=11159 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s3407:397s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:2218s;cuda3:trace_context:trace_bank__n2__Random-128-b__llama31_4b__wikics__s3407:17s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:6453/24564MiB:46%260812-15:36:41 (UTC+8)
1752tasks total=14270 running=3 done=3107 failed=0 blocked=0 pending=11160 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s3407:216s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:2036s;cuda3:trace_context:trace_bank__n2__Random-128-a__llama31_4b__wikics__s3407:572s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:99%260812-15:33:40 (UTC+8)
1751tasks total=14270 running=3 done=3107 failed=0 blocked=0 pending=11160 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s3407:34s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:1855s;cuda3:trace_context:trace_bank__n2__Random-128-a__llama31_4b__wikics__s3407:391s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:98%260812-15:30:39 (UTC+8)
1750tasks total=14270 running=3 done=3106 failed=0 blocked=0 pending=11161 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s3407:1312s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:1674s;cuda3:trace_context:trace_bank__n2__Random-128-a__llama31_4b__wikics__s3407:209s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:6469/24564MiB:100%260812-15:27:38 (UTC+8)
1749tasks total=14270 running=3 done=3106 failed=0 blocked=0 pending=11161 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s3407:1131s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:1492s;cuda3:trace_context:trace_bank__n2__Random-128-a__llama31_4b__wikics__s3407:28s | GPU cuda1:7851/24564MiB:96%;cuda2:5425/24564MiB:95%;cuda3:6469/24564MiB:99%260812-15:24:36 (UTC+8)
1748tasks total=14270 running=3 done=3104 failed=0 blocked=0 pending=11163 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s3407:950s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:1311s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:2158s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:98%260812-15:21:35 (UTC+8)
1747tasks total=14270 running=3 done=3104 failed=0 blocked=0 pending=11163 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s3407:769s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:1130s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:1977s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-15:18:34 (UTC+8)
1746tasks total=14270 running=3 done=3104 failed=0 blocked=0 pending=11163 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s3407:588s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:949s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:1796s | GPU cuda1:7851/24564MiB:100%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:98%260812-15:15:33 (UTC+8)
1745tasks total=14270 running=3 done=3104 failed=0 blocked=0 pending=11163 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s3407:404s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:765s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:1612s | GPU cuda1:7851/24564MiB:98%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:96%260812-15:12:32 (UTC+8)
1744tasks total=14270 running=3 done=3104 failed=0 blocked=0 pending=11163 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s3407:223s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:584s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:1431s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-15:09:28 (UTC+8)
1743tasks total=14270 running=3 done=3104 failed=0 blocked=0 pending=11163 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s3407:42s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:403s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:1250s | GPU cuda1:7851/24564MiB:0%;cuda2:5425/24564MiB:22%;cuda3:7369/24564MiB:100%260812-15:06:27 (UTC+8)
1742tasks total=14270 running=3 done=3102 failed=0 blocked=0 pending=11165 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s3407:918s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:222s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:1069s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:98%260812-15:03:26 (UTC+8)
1741tasks total=14270 running=3 done=3102 failed=0 blocked=0 pending=11165 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s3407:738s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__pubmed__s42:41s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:888s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:95%260812-15:00:24 (UTC+8)
1740tasks total=14270 running=3 done=3101 failed=0 blocked=0 pending=11166 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s3407:556s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:2436s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:706s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260812-14:57:24 (UTC+8)
1739tasks total=14270 running=3 done=3101 failed=0 blocked=0 pending=11166 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s3407:375s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:2255s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:525s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:94%;cuda3:7369/24564MiB:100%260812-14:54:22 (UTC+8)
1738tasks total=14270 running=3 done=3101 failed=0 blocked=0 pending=11166 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s3407:193s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:2073s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:343s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-14:51:21 (UTC+8)
1737tasks total=14270 running=3 done=3101 failed=0 blocked=0 pending=11166 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s3407:12s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:1892s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s3407:162s | GPU cuda1:7663/24564MiB:26%;cuda2:5425/24564MiB:95%;cuda3:7349/24564MiB:98%260812-14:48:19 (UTC+8)
1736tasks total=14270 running=3 done=3099 failed=0 blocked=0 pending=11168 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s3407:884s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:1709s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:2287s | GPU cuda1:7851/24564MiB:98%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-14:45:18 (UTC+8)
1735tasks total=14270 running=3 done=3099 failed=0 blocked=0 pending=11168 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s3407:703s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:1528s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:2106s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260812-14:42:15 (UTC+8)
1734tasks total=14270 running=3 done=3099 failed=0 blocked=0 pending=11168 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s3407:521s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:1345s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:1924s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-14:39:14 (UTC+8)
1733tasks total=14270 running=3 done=3099 failed=0 blocked=0 pending=11168 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s3407:340s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:1164s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:1743s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:96%260812-14:36:11 (UTC+8)
1732tasks total=14270 running=3 done=3099 failed=0 blocked=0 pending=11168 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s3407:159s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:983s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:1562s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-14:33:10 (UTC+8)
1731tasks total=14270 running=3 done=3097 failed=0 blocked=0 pending=11170 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s3407:854s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:799s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:1377s | GPU cuda1:7851/24564MiB:95%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260812-14:30:09 (UTC+8)
1730tasks total=14270 running=3 done=3097 failed=0 blocked=0 pending=11170 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s3407:673s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:618s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:1196s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:95%;cuda3:7369/24564MiB:100%260812-14:27:05 (UTC+8)
1729tasks total=14270 running=3 done=3097 failed=0 blocked=0 pending=11170 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s3407:492s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:436s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:1015s | GPU cuda1:7851/24564MiB:98%;cuda2:5425/24564MiB:100%;cuda3:7369/24564MiB:100%260812-14:24:04 (UTC+8)
1728tasks total=14270 running=3 done=3097 failed=0 blocked=0 pending=11170 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s3407:311s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:256s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:834s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:94%;cuda3:7369/24564MiB:100%260812-14:21:02 (UTC+8)
1727tasks total=14270 running=3 done=3097 failed=0 blocked=0 pending=11170 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s3407:130s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__pubmed__s42:75s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:653s | GPU cuda1:7851/24564MiB:97%;cuda2:5425/24564MiB:94%;cuda3:7369/24564MiB:100%260812-14:18:02 (UTC+8)
1726tasks total=14270 running=3 done=3093 failed=0 blocked=0 pending=11174 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s3407:826s;cuda2:trace_context:trace_bank__base__Full-256-b__llama31_4b__pubmed__s42:895s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:472s | GPU cuda1:7851/24564MiB:95%;cuda2:4585/24564MiB:97%;cuda3:7369/24564MiB:100%260812-14:15:01 (UTC+8)
1725tasks total=14270 running=3 done=3093 failed=0 blocked=0 pending=11174 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s3407:643s;cuda2:trace_context:trace_bank__base__Full-256-b__llama31_4b__pubmed__s42:711s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:289s | GPU cuda1:7851/24564MiB:98%;cuda2:4585/24564MiB:25%;cuda3:7369/24564MiB:100%260812-14:12:00 (UTC+8)
1724tasks total=14270 running=3 done=3093 failed=0 blocked=0 pending=11174 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s3407:462s;cuda2:trace_context:trace_bank__base__Full-256-b__llama31_4b__pubmed__s42:530s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s3407:108s | GPU cuda1:7851/24564MiB:98%;cuda2:4585/24564MiB:97%;cuda3:7349/24564MiB:100%260812-14:08:56 (UTC+8)
1723tasks total=14270 running=3 done=3091 failed=0 blocked=0 pending=11176 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s3407:281s;cuda2:trace_context:trace_bank__base__Full-256-b__llama31_4b__pubmed__s42:350s;cuda3:trace_context:trace_bank__n2__Full-256-b__llama31_4b__wikics__s3407:862s | GPU cuda1:7851/24564MiB:97%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:98%260812-14:05:56 (UTC+8)
1722tasks total=14270 running=3 done=3091 failed=0 blocked=0 pending=11176 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s3407:100s;cuda2:trace_context:trace_bank__base__Full-256-b__llama31_4b__pubmed__s42:168s;cuda3:trace_context:trace_bank__n2__Full-256-b__llama31_4b__wikics__s3407:680s | GPU cuda1:7851/24564MiB:95%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:99%260812-14:02:55 (UTC+8)
1721tasks total=14270 running=3 done=3088 failed=0 blocked=0 pending=11179 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:2248s;cuda2:trace_context:trace_bank__base__Full-256-a__llama31_4b__pubmed__s42:983s;cuda3:trace_context:trace_bank__n2__Full-256-b__llama31_4b__wikics__s3407:499s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:98%260812-13:59:53 (UTC+8)
1720tasks total=14270 running=3 done=3088 failed=0 blocked=0 pending=11179 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:2067s;cuda2:trace_context:trace_bank__base__Full-256-a__llama31_4b__pubmed__s42:802s;cuda3:trace_context:trace_bank__n2__Full-256-b__llama31_4b__wikics__s3407:317s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-13:56:52 (UTC+8)
1719tasks total=14270 running=3 done=3088 failed=0 blocked=0 pending=11179 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:1886s;cuda2:trace_context:trace_bank__base__Full-256-a__llama31_4b__pubmed__s42:621s;cuda3:trace_context:trace_bank__n2__Full-256-b__llama31_4b__wikics__s3407:137s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:97%260812-13:53:50 (UTC+8)
1718tasks total=14270 running=3 done=3087 failed=0 blocked=0 pending=11180 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:1705s;cuda2:trace_context:trace_bank__base__Full-256-a__llama31_4b__pubmed__s42:440s;cuda3:trace_context:trace_bank__n2__Full-256-a__llama31_4b__wikics__s3407:887s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:98%260812-13:50:49 (UTC+8)
1717tasks total=14270 running=3 done=3087 failed=0 blocked=0 pending=11180 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:1524s;cuda2:trace_context:trace_bank__base__Full-256-a__llama31_4b__pubmed__s42:259s;cuda3:trace_context:trace_bank__n2__Full-256-a__llama31_4b__wikics__s3407:706s | GPU cuda1:7851/24564MiB:95%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-13:47:49 (UTC+8)
1716tasks total=14270 running=3 done=3087 failed=0 blocked=0 pending=11180 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:1343s;cuda2:trace_context:trace_bank__base__Full-256-a__llama31_4b__pubmed__s42:78s;cuda3:trace_context:trace_bank__n2__Full-256-a__llama31_4b__wikics__s3407:525s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-13:44:47 (UTC+8)
1715tasks total=14270 running=3 done=3084 failed=0 blocked=0 pending=11183 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:1162s;cuda2:trace_context:trace_bank__base__Full-128-b__llama31_4b__pubmed__s42:685s;cuda3:trace_context:trace_bank__n2__Full-256-a__llama31_4b__wikics__s3407:344s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:97%;cuda3:6529/24564MiB:98%260812-13:41:46 (UTC+8)
1714tasks total=14270 running=3 done=3084 failed=0 blocked=0 pending=11183 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:981s;cuda2:trace_context:trace_bank__base__Full-128-b__llama31_4b__pubmed__s42:504s;cuda3:trace_context:trace_bank__n2__Full-256-a__llama31_4b__wikics__s3407:163s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:95%;cuda3:6529/24564MiB:100%260812-13:38:45 (UTC+8)
1713tasks total=14270 running=3 done=3082 failed=0 blocked=0 pending=11185 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:800s;cuda2:trace_context:trace_bank__base__Full-128-b__llama31_4b__pubmed__s42:322s;cuda3:trace_context:trace_bank__n2__Full-128-b__llama31_4b__wikics__s3407:720s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:95%;cuda3:6469/24564MiB:99%260812-13:35:44 (UTC+8)
1712tasks total=14270 running=3 done=3082 failed=0 blocked=0 pending=11185 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:619s;cuda2:trace_context:trace_bank__base__Full-128-b__llama31_4b__pubmed__s42:141s;cuda3:trace_context:trace_bank__n2__Full-128-b__llama31_4b__wikics__s3407:539s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:97%;cuda3:6469/24564MiB:99%260812-13:32:43 (UTC+8)
1711tasks total=14270 running=3 done=3081 failed=0 blocked=0 pending=11186 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:433s;cuda2:trace_context:trace_bank__base__Full-128-a__llama31_4b__pubmed__s42:737s;cuda3:trace_context:trace_bank__n2__Full-128-b__llama31_4b__wikics__s3407:354s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:97%;cuda3:6469/24564MiB:98%260812-13:29:42 (UTC+8)
1710tasks total=14270 running=3 done=3081 failed=0 blocked=0 pending=11186 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:252s;cuda2:trace_context:trace_bank__base__Full-128-a__llama31_4b__pubmed__s42:556s;cuda3:trace_context:trace_bank__n2__Full-128-b__llama31_4b__wikics__s3407:173s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:95%;cuda3:6469/24564MiB:98%260812-13:26:36 (UTC+8)
1709tasks total=14270 running=3 done=3080 failed=0 blocked=0 pending=11187 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s3407:71s;cuda2:trace_context:trace_bank__base__Full-128-a__llama31_4b__pubmed__s42:375s;cuda3:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s3407:728s | GPU cuda1:7851/24564MiB:99%;cuda2:4565/24564MiB:95%;cuda3:6469/24564MiB:98%260812-13:23:35 (UTC+8)
1708tasks total=14270 running=3 done=3079 failed=0 blocked=0 pending=11188 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:2218s;cuda2:trace_context:trace_bank__base__Full-128-a__llama31_4b__pubmed__s42:194s;cuda3:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s3407:548s | GPU cuda1:7851/24564MiB:99%;cuda2:4565/24564MiB:95%;cuda3:6469/24564MiB:99%260812-13:20:34 (UTC+8)
1707tasks total=14270 running=3 done=3079 failed=0 blocked=0 pending=11188 | results done=1051/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:2035s;cuda2:trace_context:trace_bank__base__Full-128-a__llama31_4b__pubmed__s42:12s;cuda3:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s3407:366s | GPU cuda1:7851/24564MiB:100%;cuda2:4445/24564MiB:12%;cuda3:6469/24564MiB:98%260812-13:17:33 (UTC+8)
1706tasks total=14270 running=3 done=3070 failed=0 blocked=0 pending=11197 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:1855s;cuda2:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__pubmed__s42:948s;cuda3:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s3407:185s | GPU cuda1:7851/24564MiB:95%;cuda2:4585/24564MiB:97%;cuda3:6469/24564MiB:100%260812-13:14:31 (UTC+8)
1705tasks total=14270 running=3 done=3070 failed=0 blocked=0 pending=11197 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:1673s;cuda2:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__pubmed__s42:766s;cuda3:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s3407:3s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:97%;cuda3:2305/24564MiB:47%260812-13:11:30 (UTC+8)
1704tasks total=14270 running=3 done=3068 failed=0 blocked=0 pending=11199 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:1492s;cuda2:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__pubmed__s42:585s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s3407:781s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:98%260812-13:08:29 (UTC+8)
1703tasks total=14270 running=3 done=3068 failed=0 blocked=0 pending=11199 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:1312s;cuda2:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__pubmed__s42:405s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s3407:600s | GPU cuda1:7851/24564MiB:29%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:98%260812-13:05:28 (UTC+8)
1702tasks total=14270 running=3 done=3068 failed=0 blocked=0 pending=11199 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:1131s;cuda2:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__pubmed__s42:223s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s3407:419s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:98%260812-13:02:27 (UTC+8)
1701tasks total=14270 running=3 done=3068 failed=0 blocked=0 pending=11199 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:950s;cuda2:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__pubmed__s42:43s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s3407:238s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-12:59:26 (UTC+8)
1700tasks total=14270 running=3 done=3067 failed=0 blocked=0 pending=11200 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:768s;cuda2:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__pubmed__s42:880s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s3407:57s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:99%260812-12:56:25 (UTC+8)
1699tasks total=14270 running=3 done=3066 failed=0 blocked=0 pending=11201 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:587s;cuda2:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__pubmed__s42:699s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s3407:835s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:98%260812-12:53:24 (UTC+8)
1698tasks total=14270 running=3 done=3066 failed=0 blocked=0 pending=11201 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:407s;cuda2:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__pubmed__s42:518s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s3407:654s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-12:50:23 (UTC+8)
1697tasks total=14270 running=3 done=3066 failed=0 blocked=0 pending=11201 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:225s;cuda2:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__pubmed__s42:337s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s3407:473s | GPU cuda1:7851/24564MiB:96%;cuda2:4585/24564MiB:97%;cuda3:6529/24564MiB:100%260812-12:47:22 (UTC+8)
1696tasks total=14270 running=3 done=3066 failed=0 blocked=0 pending=11201 | results done=1045/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s3407:45s;cuda2:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__pubmed__s42:156s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s3407:292s | GPU cuda1:7851/24564MiB:100%;cuda2:4585/24564MiB:95%;cuda3:6529/24564MiB:99%260812-12:44:21 (UTC+8)
1695tasks total=14270 running=3 done=3061 failed=0 blocked=0 pending=11206 | results done=1043/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s3407:921s;cuda2:fixed_downstream:run__T1_FixedFull512__llama31_4b__pubmed__s42__sage2:9s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s3407:111s | GPU cuda1:7851/24564MiB:97%;cuda2:895/24564MiB:55%;cuda3:6529/24564MiB:100%260812-12:41:20 (UTC+8)
1694tasks total=14270 running=3 done=3053 failed=0 blocked=0 pending=11214 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s3407:739s;cuda2:trace_context:trace_bank__base__Full-512-b__llama31_4b__pubmed__s42:1315s;cuda3:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s3407:1259s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:95%;cuda3:6789/24564MiB:100%260812-12:38:19 (UTC+8)
1693tasks total=14270 running=3 done=3053 failed=0 blocked=0 pending=11214 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s3407:558s;cuda2:trace_context:trace_bank__base__Full-512-b__llama31_4b__pubmed__s42:1134s;cuda3:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s3407:1078s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:100%;cuda3:6789/24564MiB:92%260812-12:35:17 (UTC+8)
1692tasks total=14270 running=3 done=3053 failed=0 blocked=0 pending=11214 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s3407:377s;cuda2:trace_context:trace_bank__base__Full-512-b__llama31_4b__pubmed__s42:953s;cuda3:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s3407:897s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:99%260812-12:32:16 (UTC+8)
1691tasks total=14270 running=3 done=3053 failed=0 blocked=0 pending=11214 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s3407:196s;cuda2:trace_context:trace_bank__base__Full-512-b__llama31_4b__pubmed__s42:772s;cuda3:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s3407:716s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:100%;cuda3:6789/24564MiB:97%260812-12:29:15 (UTC+8)
1690tasks total=14270 running=3 done=3053 failed=0 blocked=0 pending=11214 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s3407:15s;cuda2:trace_context:trace_bank__base__Full-512-b__llama31_4b__pubmed__s42:591s;cuda3:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s3407:535s | GPU cuda1:7685/24564MiB:25%;cuda2:4885/24564MiB:96%;cuda3:6789/24564MiB:100%260812-12:26:14 (UTC+8)
1689tasks total=14270 running=3 done=3052 failed=0 blocked=0 pending=11215 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s3407:889s;cuda2:trace_context:trace_bank__base__Full-512-b__llama31_4b__pubmed__s42:410s;cuda3:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s3407:354s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:100%;cuda3:6789/24564MiB:97%260812-12:23:13 (UTC+8)
1688tasks total=14270 running=3 done=3052 failed=0 blocked=0 pending=11215 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s3407:709s;cuda2:trace_context:trace_bank__base__Full-512-b__llama31_4b__pubmed__s42:229s;cuda3:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s3407:173s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:100%260812-12:20:12 (UTC+8)
1687tasks total=14270 running=3 done=3051 failed=0 blocked=0 pending=11216 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s3407:527s;cuda2:trace_context:trace_bank__base__Full-512-b__llama31_4b__pubmed__s42:48s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s3407:1320s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:6789/24564MiB:100%260812-12:17:11 (UTC+8)
1686tasks total=14270 running=3 done=3050 failed=0 blocked=0 pending=11217 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s3407:345s;cuda2:trace_context:trace_bank__base__Full-512-a__llama31_4b__pubmed__s42:1292s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s3407:1138s | GPU cuda1:7851/24564MiB:94%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:100%260812-12:14:10 (UTC+8)
1685tasks total=14270 running=3 done=3050 failed=0 blocked=0 pending=11217 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s3407:165s;cuda2:trace_context:trace_bank__base__Full-512-a__llama31_4b__pubmed__s42:1111s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s3407:957s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:6789/24564MiB:99%260812-12:11:08 (UTC+8)
1684tasks total=14270 running=3 done=3048 failed=0 blocked=0 pending=11219 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s3407:862s;cuda2:trace_context:trace_bank__base__Full-512-a__llama31_4b__pubmed__s42:931s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s3407:776s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:100%;cuda3:6789/24564MiB:100%260812-12:08:07 (UTC+8)
1683tasks total=14270 running=3 done=3048 failed=0 blocked=0 pending=11219 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s3407:682s;cuda2:trace_context:trace_bank__base__Full-512-a__llama31_4b__pubmed__s42:750s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s3407:595s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:97%260812-12:05:06 (UTC+8)
1682tasks total=14270 running=3 done=3048 failed=0 blocked=0 pending=11219 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s3407:501s;cuda2:trace_context:trace_bank__base__Full-512-a__llama31_4b__pubmed__s42:569s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s3407:414s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:100%;cuda3:6789/24564MiB:99%260812-12:02:06 (UTC+8)
1681tasks total=14270 running=3 done=3048 failed=0 blocked=0 pending=11219 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s3407:319s;cuda2:trace_context:trace_bank__base__Full-512-a__llama31_4b__pubmed__s42:387s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s3407:233s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:96%;cuda3:6789/24564MiB:99%260812-11:59:04 (UTC+8)
1680tasks total=14270 running=3 done=3048 failed=0 blocked=0 pending=11219 | results done=1039/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s3407:138s;cuda2:trace_context:trace_bank__base__Full-512-a__llama31_4b__pubmed__s42:206s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s3407:52s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:97%;cuda3:6789/24564MiB:100%260812-11:56:03 (UTC+8)
1679tasks total=14270 running=3 done=3045 failed=0 blocked=0 pending=11222 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s3407:834s;cuda2:trace_context:trace_bank__base__Full-512-a__llama31_4b__pubmed__s42:25s;cuda3:trace_analysis:trace_analysis__n1__llama31_4b__wikics__s3407:10s | GPU cuda1:7851/24564MiB:71%;cuda2:4885/24564MiB:100%;cuda3:3535/24564MiB:99%260812-11:53:02 (UTC+8)
1678tasks total=14270 running=3 done=3042 failed=0 blocked=0 pending=11225 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s3407:653s;cuda2:trace_context:trace_bank__base__Self__llama31_4b__pubmed__s42:531s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s3407:1471s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:97%;cuda3:7349/24564MiB:100%260812-11:50:01 (UTC+8)
1677tasks total=14270 running=3 done=3042 failed=0 blocked=0 pending=11225 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s3407:472s;cuda2:trace_context:trace_bank__base__Self__llama31_4b__pubmed__s42:350s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s3407:1290s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:98%;cuda3:7349/24564MiB:100%260812-11:47:00 (UTC+8)
1676tasks total=14270 running=3 done=3042 failed=0 blocked=0 pending=11225 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s3407:291s;cuda2:trace_context:trace_bank__base__Self__llama31_4b__pubmed__s42:169s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s3407:1109s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:95%;cuda3:7349/24564MiB:99%260812-11:43:59 (UTC+8)
1675tasks total=14270 running=3 done=3041 failed=0 blocked=0 pending=11226 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s3407:110s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:4455s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s3407:928s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:5445/24564MiB:94%260812-11:40:58 (UTC+8)
1674tasks total=14270 running=3 done=3039 failed=0 blocked=0 pending=11228 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-b__llama3_8b__wikics__s3407:1014s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:4274s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s3407:747s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:99%;cuda3:5445/24564MiB:100%260812-11:37:57 (UTC+8)
1673tasks total=14270 running=3 done=3039 failed=0 blocked=0 pending=11228 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-b__llama3_8b__wikics__s3407:833s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:4092s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s3407:565s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:5445/24564MiB:100%260812-11:34:56 (UTC+8)
1672tasks total=14270 running=3 done=3039 failed=0 blocked=0 pending=11228 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-b__llama3_8b__wikics__s3407:652s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:3912s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s3407:384s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:99%;cuda3:5445/24564MiB:93%260812-11:31:54 (UTC+8)
1671tasks total=14270 running=3 done=3039 failed=0 blocked=0 pending=11228 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-b__llama3_8b__wikics__s3407:471s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:3730s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s3407:203s | GPU cuda1:7851/24564MiB:95%;cuda2:7947/24564MiB:99%;cuda3:5445/24564MiB:95%260812-11:28:54 (UTC+8)
1670tasks total=14270 running=3 done=3039 failed=0 blocked=0 pending=11228 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-b__llama3_8b__wikics__s3407:290s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:3549s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s3407:22s | GPU cuda1:7851/24564MiB:95%;cuda2:7947/24564MiB:99%;cuda3:5425/24564MiB:100%260812-11:25:52 (UTC+8)
1669tasks total=14270 running=3 done=3038 failed=0 blocked=0 pending=11229 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-b__llama3_8b__wikics__s3407:109s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:3368s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s3407:1306s | GPU cuda1:7851/24564MiB:82%;cuda2:7947/24564MiB:100%;cuda3:5445/24564MiB:95%260812-11:22:52 (UTC+8)
1668tasks total=14270 running=3 done=3037 failed=0 blocked=0 pending=11230 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s3407:1011s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:3187s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s3407:1125s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:5445/24564MiB:100%260812-11:19:50 (UTC+8)
1667tasks total=14270 running=3 done=3037 failed=0 blocked=0 pending=11230 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s3407:830s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:3006s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s3407:944s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:5445/24564MiB:95%260812-11:16:49 (UTC+8)
1666tasks total=14270 running=3 done=3037 failed=0 blocked=0 pending=11230 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s3407:649s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:2825s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s3407:763s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:99%;cuda3:5445/24564MiB:95%260812-11:13:48 (UTC+8)
1665tasks total=14270 running=3 done=3037 failed=0 blocked=0 pending=11230 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s3407:468s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:2644s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s3407:582s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:5445/24564MiB:100%260812-11:10:47 (UTC+8)
1664tasks total=14270 running=3 done=3037 failed=0 blocked=0 pending=11230 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s3407:286s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:2462s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s3407:400s | GPU cuda1:7851/24564MiB:98%;cuda2:7947/24564MiB:95%;cuda3:5445/24564MiB:100%260812-11:07:46 (UTC+8)
1663tasks total=14270 running=3 done=3037 failed=0 blocked=0 pending=11230 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s3407:103s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:2279s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s3407:218s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:5445/24564MiB:94%260812-11:04:44 (UTC+8)
1662tasks total=14270 running=3 done=3035 failed=0 blocked=0 pending=11232 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s3407:1383s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:2097s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s3407:35s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:5425/24564MiB:100%260812-11:01:41 (UTC+8)
1661tasks total=14270 running=3 done=3033 failed=0 blocked=0 pending=11234 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s3407:1202s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:1916s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s3407:733s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:4905/24564MiB:96%260812-10:58:39 (UTC+8)
1660tasks total=14270 running=3 done=3033 failed=0 blocked=0 pending=11234 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s3407:1021s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:1735s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s3407:552s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:99%;cuda3:4905/24564MiB:96%260812-10:55:38 (UTC+8)
1659tasks total=14270 running=3 done=3033 failed=0 blocked=0 pending=11234 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s3407:840s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:1554s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s3407:371s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:99%;cuda3:4905/24564MiB:96%260812-10:52:37 (UTC+8)
1658tasks total=14270 running=3 done=3033 failed=0 blocked=0 pending=11234 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s3407:659s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:1373s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s3407:190s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:99%;cuda3:4905/24564MiB:100%260812-10:49:36 (UTC+8)
1657tasks total=14270 running=3 done=3033 failed=0 blocked=0 pending=11234 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s3407:478s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:1192s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s3407:9s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:99%;cuda3:4421/24564MiB:14%260812-10:46:35 (UTC+8)
1656tasks total=14270 running=3 done=3032 failed=0 blocked=0 pending=11235 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s3407:297s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:1011s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s3407:703s | GPU cuda1:7851/24564MiB:100%;cuda2:7947/24564MiB:100%;cuda3:4905/24564MiB:96%260812-10:43:34 (UTC+8)
1655tasks total=14270 running=3 done=3032 failed=0 blocked=0 pending=11235 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s3407:116s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:830s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s3407:522s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:99%;cuda3:4905/24564MiB:97%260812-10:40:33 (UTC+8)
1654tasks total=14270 running=3 done=3031 failed=0 blocked=0 pending=11236 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s3407:1394s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:649s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s3407:341s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:4905/24564MiB:96%260812-10:37:32 (UTC+8)
1653tasks total=14270 running=3 done=3031 failed=0 blocked=0 pending=11236 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s3407:1213s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:468s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s3407:160s | GPU cuda1:7851/24564MiB:97%;cuda2:7947/24564MiB:100%;cuda3:4905/24564MiB:97%260812-10:34:31 (UTC+8)
1652tasks total=14270 running=3 done=3029 failed=0 blocked=0 pending=11238 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s3407:1032s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:287s;cuda3:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s3407:562s | GPU cuda1:7851/24564MiB:100%;cuda2:7947/24564MiB:99%;cuda3:4605/24564MiB:97%260812-10:31:30 (UTC+8)
1651tasks total=14270 running=3 done=3029 failed=0 blocked=0 pending=11238 | results done=1038/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s3407:851s;cuda2:sft:sft__trace__llama31_4b__pubmed__s42:106s;cuda3:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s3407:381s | GPU cuda1:7851/24564MiB:97%;cuda2:7927/24564MiB:99%;cuda3:4605/24564MiB:97%260812-10:28:29 (UTC+8)
1650tasks total=14270 running=3 done=3025 failed=0 blocked=0 pending=11242 | results done=1035/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s3407:670s;cuda2:external_downstream:run__LinguGKD__llama3_8b__wikics__s3407__sage1:10s;cuda3:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s3407:200s | GPU cuda1:7851/24564MiB:99%;cuda2:1421/24564MiB:23%;cuda3:4605/24564MiB:97%260812-10:25:28 (UTC+8)
1649tasks total=14270 running=3 done=3020 failed=0 blocked=0 pending=11247 | results done=1032/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s3407:489s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:2361s;cuda3:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s3407:19s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:4565/24564MiB:97%260812-10:22:27 (UTC+8)
1648tasks total=14270 running=3 done=3019 failed=0 blocked=0 pending=11248 | results done=1032/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s3407:308s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:2180s;cuda3:trace_context:trace_bank__n1__HopQuota-256-a__llama31_4b__wikics__s3407:419s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4605/24564MiB:97%260812-10:19:26 (UTC+8)
1647tasks total=14270 running=3 done=3019 failed=0 blocked=0 pending=11248 | results done=1032/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s3407:127s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:1998s;cuda3:trace_context:trace_bank__n1__HopQuota-256-a__llama31_4b__wikics__s3407:237s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:4605/24564MiB:97%260812-10:16:25 (UTC+8)
1646tasks total=14270 running=3 done=3017 failed=0 blocked=0 pending=11250 | results done=1031/5050 | RUNNING cuda1:g4_teacher:g4_teacher__llama3_8b__wikics__s3407:697s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:1817s;cuda3:trace_context:trace_bank__n1__HopQuota-256-a__llama31_4b__wikics__s3407:56s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:4605/24564MiB:95%260812-10:13:24 (UTC+8)
1645tasks total=14270 running=3 done=3015 failed=0 blocked=0 pending=11252 | results done=1031/5050 | RUNNING cuda1:g4_teacher:g4_teacher__llama3_8b__wikics__s3407:516s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:1636s;cuda3:trace_context:trace_bank__n1__HopQuota-128-b__llama31_4b__wikics__s3407:336s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:96%;cuda3:4565/24564MiB:25%260812-10:10:23 (UTC+8)
1644tasks total=14270 running=3 done=3015 failed=0 blocked=0 pending=11252 | results done=1031/5050 | RUNNING cuda1:g4_teacher:g4_teacher__llama3_8b__wikics__s3407:335s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:1455s;cuda3:trace_context:trace_bank__n1__HopQuota-128-b__llama31_4b__wikics__s3407:154s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:96%;cuda3:4565/24564MiB:95%260812-10:07:22 (UTC+8)
1643tasks total=14270 running=3 done=3014 failed=0 blocked=0 pending=11253 | results done=1031/5050 | RUNNING cuda1:g4_teacher:g4_teacher__llama3_8b__wikics__s3407:154s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:1274s;cuda3:trace_context:trace_bank__n1__HopQuota-128-a__llama31_4b__wikics__s3407:430s | GPU cuda1:7851/24564MiB:90%;cuda2:7851/24564MiB:95%;cuda3:4565/24564MiB:97%260812-10:04:21 (UTC+8)
1642tasks total=14270 running=3 done=3012 failed=0 blocked=0 pending=11255 | results done=1030/5050 | RUNNING cuda1:trace_downstream:run__G2_CrossStudent__llama3_8b__wikics__s3407__gcn3:15s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:1093s;cuda3:trace_context:trace_bank__n1__HopQuota-128-a__llama31_4b__wikics__s3407:249s | GPU cuda1:1637/24564MiB:80%;cuda2:7851/24564MiB:100%;cuda3:4565/24564MiB:97%260812-10:01:20 (UTC+8)
1641tasks total=14270 running=3 done=3006 failed=0 blocked=0 pending=11261 | results done=1024/5050 | RUNNING cuda1:trace_downstream:run__O1_OrderA__llama3_8b__wikics__s3407__gcn3:147s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:912s;cuda3:trace_context:trace_bank__n1__HopQuota-128-a__llama31_4b__wikics__s3407:68s | GPU cuda1:1637/24564MiB:85%;cuda2:7851/24564MiB:95%;cuda3:4565/24564MiB:95%260812-09:58:18 (UTC+8)
1640tasks total=14270 running=3 done=3002 failed=0 blocked=0 pending=11265 | results done=1022/5050 | RUNNING cuda1:trace_downstream:run__I3_HiddenMSE__llama3_8b__wikics__s3407__gcn3:6s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:731s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:3840s | GPU cuda1:1673/24564MiB:82%;cuda2:7851/24564MiB:96%;cuda3:5445/24564MiB:100%260812-09:55:18 (UTC+8)
1639tasks total=14270 running=3 done=2996 failed=0 blocked=0 pending=11271 | results done=1016/5050 | RUNNING cuda1:trace_downstream:run__C6_BudgetSweep__budget_128__llama3_8b__wikics__s3407__gcn3:19s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:550s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:3659s | GPU cuda1:1637/24564MiB:80%;cuda2:7851/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:52:16 (UTC+8)
1638tasks total=14270 running=3 done=2991 failed=0 blocked=0 pending=11276 | results done=1011/5050 | RUNNING cuda1:trace_downstream:run__C5_LeaveFamilyOut__without_Full__llama3_8b__wikics__s3407__gcn3:5s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:369s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:3478s | GPU cuda1:731/24564MiB:57%;cuda2:7851/24564MiB:98%;cuda3:5445/24564MiB:0%260812-09:49:15 (UTC+8)
1637tasks total=14270 running=3 done=2985 failed=0 blocked=0 pending=11282 | results done=1005/5050 | RUNNING cuda1:trace_downstream:run__A7_SoftVsHard__hard__llama3_8b__wikics__s3407__gcn3:21s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:188s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:3297s | GPU cuda1:1637/24564MiB:79%;cuda2:7851/24564MiB:96%;cuda3:5445/24564MiB:0%260812-09:46:14 (UTC+8)
1636tasks total=14270 running=3 done=2980 failed=0 blocked=0 pending=11287 | results done=1000/5050 | RUNNING cuda1:trace_downstream:run__A3_NoOrder__llama3_8b__wikics__s3407__gcn3:7s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s3407:7s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:3116s | GPU cuda1:1637/24564MiB:85%;cuda2:7625/24564MiB:17%;cuda3:5445/24564MiB:0%260812-09:43:13 (UTC+8)
1635tasks total=14270 running=3 done=2972 failed=0 blocked=0 pending=11295 | results done=993/5050 | RUNNING cuda1:trace_downstream:run__T8_TRACEBilevelHard__llama3_8b__wikics__s3407__gcn3:1s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:6512s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:2935s | GPU cuda1:401/24564MiB:0%;cuda2:11839/24564MiB:99%;cuda3:5445/24564MiB:100%260812-09:40:12 (UTC+8)
1634tasks total=14270 running=3 done=2963 failed=0 blocked=0 pending=11304 | results done=984/5050 | RUNNING cuda1:trace_downstream:run__T6_TRACEProxy__llama3_8b__wikics__s3407__sage3:15s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:6331s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:2754s | GPU cuda1:1441/24564MiB:75%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:95%260812-09:37:11 (UTC+8)
1633tasks total=14270 running=3 done=2954 failed=0 blocked=0 pending=11313 | results done=975/5050 | RUNNING cuda1:trace_downstream:run__T5_RealizabilityOnly__llama3_8b__wikics__s3407__gcn3:18s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:6150s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:2573s | GPU cuda1:1637/24564MiB:79%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:34:10 (UTC+8)
1632tasks total=14270 running=3 done=2950 failed=0 blocked=0 pending=11317 | results done=971/5050 | RUNNING cuda1:trace_downstream:run__T4_StudentOptGlobal__llama3_8b__wikics__s3407__sage2:86s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:5969s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:2392s | GPU cuda1:1435/24564MiB:71%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:31:09 (UTC+8)
1631tasks total=14270 running=3 done=2948 failed=0 blocked=0 pending=11319 | results done=969/5050 | RUNNING cuda1:trace_downstream:run__T4_StudentOptGlobal__llama3_8b__wikics__s3407__gcn3:127s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:5788s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:2212s | GPU cuda1:1637/24564MiB:83%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:28:08 (UTC+8)
1630tasks total=14270 running=3 done=2947 failed=0 blocked=0 pending=11320 | results done=968/5050 | RUNNING cuda1:trace_downstream:run__T4_StudentOptGlobal__llama3_8b__wikics__s3407__gcn2:63s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:5606s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:2029s | GPU cuda1:1553/24564MiB:77%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:100%260812-09:25:08 (UTC+8)
1629tasks total=14270 running=3 done=2943 failed=0 blocked=0 pending=11324 | results done=964/5050 | RUNNING cuda1:trace_downstream:run__T3_TeacherOptGlobal__llama3_8b__wikics__s3407__sage1:5s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:5425s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:1848s | GPU cuda1:1051/24564MiB:64%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:22:05 (UTC+8)
1628tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:2121s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:5244s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:1667s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:19:04 (UTC+8)
1627tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:1940s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:5063s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:1486s | GPU cuda1:7851/24564MiB:100%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:16:03 (UTC+8)
1626tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:1759s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:4882s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:1305s | GPU cuda1:7851/24564MiB:100%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:100%260812-09:13:02 (UTC+8)
1625tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:1578s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:4701s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:1124s | GPU cuda1:7851/24564MiB:97%;cuda2:11839/24564MiB:98%;cuda3:5445/24564MiB:95%260812-09:10:01 (UTC+8)
1624tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:1397s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:4520s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:943s | GPU cuda1:7851/24564MiB:0%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:07:00 (UTC+8)
1623tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:1216s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:4338s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:762s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:03:59 (UTC+8)
1622tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:1034s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:4157s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:580s | GPU cuda1:7851/24564MiB:97%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-09:00:58 (UTC+8)
1621tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:853s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:3976s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:399s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:100%260812-08:57:56 (UTC+8)
1620tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:673s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:3795s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:219s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:94%260812-08:54:55 (UTC+8)
1619tasks total=14270 running=3 done=2936 failed=0 blocked=0 pending=11331 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:492s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:3615s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s3407:38s | GPU cuda1:7851/24564MiB:0%;cuda2:11839/24564MiB:100%;cuda3:4549/24564MiB:0%260812-08:51:55 (UTC+8)
1618tasks total=14270 running=3 done=2935 failed=0 blocked=0 pending=11332 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:310s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:3433s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:3907s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-08:48:54 (UTC+8)
1617tasks total=14270 running=3 done=2935 failed=0 blocked=0 pending=11332 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s3407:129s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:3252s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:3725s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-08:45:52 (UTC+8)
1616tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:2103s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:3071s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:3544s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-08:42:51 (UTC+8)
1615tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:1922s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:2890s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:3363s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-08:39:50 (UTC+8)
1614tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:1741s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:2709s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:3182s | GPU cuda1:7851/24564MiB:100%;cuda2:11839/24564MiB:99%;cuda3:5445/24564MiB:0%260812-08:36:49 (UTC+8)
1613tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:1559s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:2528s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:3001s | GPU cuda1:7851/24564MiB:0%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:100%260812-08:33:48 (UTC+8)
1612tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:1379s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:2347s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:2820s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:99%;cuda3:5445/24564MiB:100%260812-08:30:47 (UTC+8)
1611tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:1198s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:2166s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:2639s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:99%;cuda3:5445/24564MiB:100%260812-08:27:46 (UTC+8)
1610tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:1017s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:1985s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:2459s | GPU cuda1:7851/24564MiB:100%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:94%260812-08:24:45 (UTC+8)
1609tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:836s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:1804s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:2278s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-08:21:44 (UTC+8)
1608tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:655s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:1623s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:2096s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-08:18:44 (UTC+8)
1607tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:474s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:1442s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:1916s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-08:15:42 (UTC+8)
1606tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:293s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:1261s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:1734s | GPU cuda1:7851/24564MiB:100%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-08:12:42 (UTC+8)
1605tasks total=14270 running=3 done=2934 failed=0 blocked=0 pending=11333 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s3407:111s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:1080s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:1553s | GPU cuda1:7851/24564MiB:100%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:100%260812-08:09:40 (UTC+8)
1604tasks total=14270 running=3 done=2931 failed=0 blocked=0 pending=11336 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s3407:1275s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:899s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:1372s | GPU cuda1:7851/24564MiB:33%;cuda2:11839/24564MiB:99%;cuda3:5445/24564MiB:100%260812-08:06:39 (UTC+8)
1603tasks total=14270 running=3 done=2931 failed=0 blocked=0 pending=11336 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s3407:1094s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:718s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:1191s | GPU cuda1:7851/24564MiB:97%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:100%260812-08:03:38 (UTC+8)
1602tasks total=14270 running=3 done=2931 failed=0 blocked=0 pending=11336 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s3407:913s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:537s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:1010s | GPU cuda1:7851/24564MiB:97%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-08:00:37 (UTC+8)
1601tasks total=14270 running=3 done=2931 failed=0 blocked=0 pending=11336 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s3407:732s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:356s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:829s | GPU cuda1:7851/24564MiB:97%;cuda2:11839/24564MiB:100%;cuda3:5445/24564MiB:0%260812-07:57:36 (UTC+8)
1600tasks total=14270 running=3 done=2931 failed=0 blocked=0 pending=11336 | results done=961/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s3407:551s;cuda2:sft:sft__lingu__llama3_8b__wikics__s3407:175s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:648s | GPU cuda1:7851/24564MiB:97%;cuda2:11817/24564MiB:100%;cuda3:5445/24564MiB:0%260812-07:54:35 (UTC+8)
1599tasks total=14270 running=3 done=2929 failed=0 blocked=0 pending=11338 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s3407:370s;cuda2:trace_downstream:run__N2_ShuffledText__llama3_8b__wikics__s999__gcn3:18s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:467s | GPU cuda1:7851/24564MiB:97%;cuda2:1637/24564MiB:79%;cuda3:5445/24564MiB:0%260812-07:51:34 (UTC+8)
1598tasks total=14270 running=3 done=2926 failed=0 blocked=0 pending=11341 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s3407:189s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:2184s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:286s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:5445/24564MiB:0%260812-07:48:33 (UTC+8)
1597tasks total=14270 running=3 done=2926 failed=0 blocked=0 pending=11341 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s3407:8s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:2003s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s3407:105s | GPU cuda1:7633/24564MiB:23%;cuda2:7851/24564MiB:96%;cuda3:5445/24564MiB:0%260812-07:45:32 (UTC+8)
1596tasks total=14270 running=3 done=2923 failed=0 blocked=0 pending=11344 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s3407:1166s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:1822s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:3252s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-07:42:31 (UTC+8)
1595tasks total=14270 running=3 done=2923 failed=0 blocked=0 pending=11344 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s3407:985s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:1641s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:3071s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-07:39:30 (UTC+8)
1594tasks total=14270 running=3 done=2923 failed=0 blocked=0 pending=11344 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s3407:804s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:1459s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:2889s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:99%;cuda3:4885/24564MiB:0%260812-07:36:28 (UTC+8)
1593tasks total=14270 running=3 done=2923 failed=0 blocked=0 pending=11344 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s3407:622s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:1278s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:2708s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-07:33:27 (UTC+8)
1592tasks total=14270 running=3 done=2923 failed=0 blocked=0 pending=11344 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s3407:442s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:1097s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:2527s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-07:30:26 (UTC+8)
1591tasks total=14270 running=3 done=2923 failed=0 blocked=0 pending=11344 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s3407:261s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:916s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:2346s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:4885/24564MiB:0%260812-07:27:25 (UTC+8)
1590tasks total=14270 running=3 done=2923 failed=0 blocked=0 pending=11344 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s3407:80s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:736s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:2166s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:96%;cuda3:4885/24564MiB:0%260812-07:24:24 (UTC+8)
1589tasks total=14270 running=3 done=2920 failed=0 blocked=0 pending=11347 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-b__llama3_8b__wikics__s3407:820s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:554s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:1984s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-07:21:23 (UTC+8)
1588tasks total=14270 running=3 done=2920 failed=0 blocked=0 pending=11347 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-b__llama3_8b__wikics__s3407:639s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:373s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:1803s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:4885/24564MiB:0%260812-07:18:22 (UTC+8)
1587tasks total=14270 running=3 done=2920 failed=0 blocked=0 pending=11347 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-b__llama3_8b__wikics__s3407:458s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:192s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:1622s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-07:15:21 (UTC+8)
1586tasks total=14270 running=3 done=2920 failed=0 blocked=0 pending=11347 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-b__llama3_8b__wikics__s3407:277s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s999:11s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:1441s | GPU cuda1:7851/24564MiB:97%;cuda2:7655/24564MiB:11%;cuda3:4885/24564MiB:0%260812-07:12:20 (UTC+8)
1585tasks total=14270 running=3 done=2919 failed=0 blocked=0 pending=11348 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-b__llama3_8b__wikics__s3407:96s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:2106s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:1260s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:4885/24564MiB:0%260812-07:09:19 (UTC+8)
1584tasks total=14270 running=3 done=2918 failed=0 blocked=0 pending=11349 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s3407:832s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:1925s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:1079s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:0%260812-07:06:18 (UTC+8)
1583tasks total=14270 running=3 done=2918 failed=0 blocked=0 pending=11349 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s3407:651s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:1744s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:898s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:96%;cuda3:4885/24564MiB:0%260812-07:03:17 (UTC+8)
1582tasks total=14270 running=3 done=2918 failed=0 blocked=0 pending=11349 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s3407:470s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:1563s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:717s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-07:00:16 (UTC+8)
1581tasks total=14270 running=3 done=2918 failed=0 blocked=0 pending=11349 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s3407:289s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:1382s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:536s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:96%;cuda3:4885/24564MiB:0%260812-06:57:15 (UTC+8)
1580tasks total=14270 running=3 done=2918 failed=0 blocked=0 pending=11349 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s3407:108s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:1201s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:355s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-06:54:14 (UTC+8)
1579tasks total=14270 running=3 done=2915 failed=0 blocked=0 pending=11352 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s3407:719s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:1020s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s3407:174s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:4885/24564MiB:0%260812-06:51:13 (UTC+8)
1578tasks total=14270 running=3 done=2914 failed=0 blocked=0 pending=11353 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s3407:538s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:839s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:3358s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-06:48:12 (UTC+8)
1577tasks total=14270 running=3 done=2914 failed=0 blocked=0 pending=11353 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s3407:357s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:658s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:3178s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-06:45:11 (UTC+8)
1576tasks total=14270 running=3 done=2914 failed=0 blocked=0 pending=11353 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s3407:176s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:477s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:2996s | GPU cuda1:7851/24564MiB:3%;cuda2:7851/24564MiB:96%;cuda3:4885/24564MiB:0%260812-06:42:10 (UTC+8)
1575tasks total=14270 running=3 done=2913 failed=0 blocked=0 pending=11354 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-a__llama3_8b__wikics__s3407:784s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:296s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:2815s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-06:39:09 (UTC+8)
1574tasks total=14270 running=3 done=2913 failed=0 blocked=0 pending=11354 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-a__llama3_8b__wikics__s3407:603s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s999:115s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:2635s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:0%260812-06:36:08 (UTC+8)
1573tasks total=14270 running=3 done=2911 failed=0 blocked=0 pending=11356 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-a__llama3_8b__wikics__s3407:422s;cuda2:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s999:1349s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:2454s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:100%260812-06:33:07 (UTC+8)
1572tasks total=14270 running=3 done=2911 failed=0 blocked=0 pending=11356 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-a__llama3_8b__wikics__s3407:241s;cuda2:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s999:1168s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:2273s | GPU cuda1:7851/24564MiB:89%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:96%260812-06:30:06 (UTC+8)
1571tasks total=14270 running=3 done=2911 failed=0 blocked=0 pending=11356 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-a__llama3_8b__wikics__s3407:60s;cuda2:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s999:987s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:2092s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:96%260812-06:27:05 (UTC+8)
1570tasks total=14270 running=3 done=2908 failed=0 blocked=0 pending=11359 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:4317s;cuda2:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s999:806s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:1911s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:97%260812-06:24:04 (UTC+8)
1569tasks total=14270 running=3 done=2908 failed=0 blocked=0 pending=11359 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:4136s;cuda2:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s999:625s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:1730s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:100%260812-06:21:04 (UTC+8)
1568tasks total=14270 running=3 done=2908 failed=0 blocked=0 pending=11359 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:3955s;cuda2:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s999:444s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:1549s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:100%260812-06:18:03 (UTC+8)
1567tasks total=14270 running=3 done=2908 failed=0 blocked=0 pending=11359 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:3774s;cuda2:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s999:263s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:1368s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:100%260812-06:15:02 (UTC+8)
1566tasks total=14270 running=3 done=2908 failed=0 blocked=0 pending=11359 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:3593s;cuda2:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s999:82s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:1187s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:96%260812-06:12:01 (UTC+8)
1565tasks total=14270 running=3 done=2907 failed=0 blocked=0 pending=11360 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:3412s;cuda2:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s999:1312s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:1006s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:96%260812-06:09:00 (UTC+8)
1564tasks total=14270 running=3 done=2907 failed=0 blocked=0 pending=11360 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:3231s;cuda2:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s999:1131s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:825s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:100%260812-06:05:59 (UTC+8)
1563tasks total=14270 running=3 done=2907 failed=0 blocked=0 pending=11360 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:3050s;cuda2:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s999:950s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:644s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:0%260812-06:02:58 (UTC+8)
1562tasks total=14270 running=3 done=2907 failed=0 blocked=0 pending=11360 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:2869s;cuda2:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s999:769s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:463s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:0%260812-05:59:57 (UTC+8)
1561tasks total=14270 running=3 done=2907 failed=0 blocked=0 pending=11360 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:2688s;cuda2:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s999:587s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:282s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:0%260812-05:56:56 (UTC+8)
1560tasks total=14270 running=3 done=2907 failed=0 blocked=0 pending=11360 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:2507s;cuda2:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s999:407s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s3407:101s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:0%260812-05:53:54 (UTC+8)
1559tasks total=14270 running=3 done=2905 failed=0 blocked=0 pending=11362 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:2326s;cuda2:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s999:226s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:2975s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-05:50:53 (UTC+8)
1558tasks total=14270 running=3 done=2905 failed=0 blocked=0 pending=11362 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:2145s;cuda2:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s999:45s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:2794s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-05:47:53 (UTC+8)
1557tasks total=14270 running=3 done=2903 failed=0 blocked=0 pending=11364 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:1964s;cuda2:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s999:846s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:2613s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-05:44:52 (UTC+8)
1556tasks total=14270 running=3 done=2903 failed=0 blocked=0 pending=11364 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:1784s;cuda2:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s999:665s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:2433s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:96%;cuda3:4625/24564MiB:3%260812-05:41:51 (UTC+8)
1555tasks total=14270 running=3 done=2903 failed=0 blocked=0 pending=11364 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:1603s;cuda2:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s999:484s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:2252s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-05:38:50 (UTC+8)
1554tasks total=14270 running=3 done=2903 failed=0 blocked=0 pending=11364 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:1422s;cuda2:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s999:303s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:2071s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-05:35:49 (UTC+8)
1553tasks total=14270 running=3 done=2903 failed=0 blocked=0 pending=11364 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:1241s;cuda2:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s999:122s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:1890s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-05:32:48 (UTC+8)
1552tasks total=14270 running=3 done=2902 failed=0 blocked=0 pending=11365 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:1060s;cuda2:trace_context:trace_bank__n2__HopQuota-256-a__llama3_8b__wikics__s999:919s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:1709s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-05:29:47 (UTC+8)
1551tasks total=14270 running=3 done=2902 failed=0 blocked=0 pending=11365 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:879s;cuda2:trace_context:trace_bank__n2__HopQuota-256-a__llama3_8b__wikics__s999:738s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:1528s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-05:26:46 (UTC+8)
1550tasks total=14270 running=3 done=2902 failed=0 blocked=0 pending=11365 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:698s;cuda2:trace_context:trace_bank__n2__HopQuota-256-a__llama3_8b__wikics__s999:557s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:1347s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:98%;cuda3:4625/24564MiB:0%260812-05:23:45 (UTC+8)
1549tasks total=14270 running=3 done=2902 failed=0 blocked=0 pending=11365 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:517s;cuda2:trace_context:trace_bank__n2__HopQuota-256-a__llama3_8b__wikics__s999:376s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:1166s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:4625/24564MiB:0%260812-05:20:44 (UTC+8)
1548tasks total=14270 running=3 done=2902 failed=0 blocked=0 pending=11365 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:336s;cuda2:trace_context:trace_bank__n2__HopQuota-256-a__llama3_8b__wikics__s999:195s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:985s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:95%;cuda3:4625/24564MiB:0%260812-05:17:43 (UTC+8)
1547tasks total=14270 running=3 done=2902 failed=0 blocked=0 pending=11365 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s3407:155s;cuda2:trace_context:trace_bank__n2__HopQuota-256-a__llama3_8b__wikics__s999:14s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:804s | GPU cuda1:7851/24564MiB:100%;cuda2:7681/24564MiB:13%;cuda3:4625/24564MiB:0%260812-05:14:43 (UTC+8)
1546tasks total=14270 running=3 done=2899 failed=0 blocked=0 pending=11368 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:4439s;cuda2:trace_context:trace_bank__n2__HopQuota-128-b__llama3_8b__wikics__s999:660s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:623s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:95%;cuda3:4625/24564MiB:0%260812-05:11:42 (UTC+8)
1545tasks total=14270 running=3 done=2899 failed=0 blocked=0 pending=11368 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:4259s;cuda2:trace_context:trace_bank__n2__HopQuota-128-b__llama3_8b__wikics__s999:479s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:442s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:98%;cuda3:4625/24564MiB:97%260812-05:08:41 (UTC+8)
1544tasks total=14270 running=3 done=2899 failed=0 blocked=0 pending=11368 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:4078s;cuda2:trace_context:trace_bank__n2__HopQuota-128-b__llama3_8b__wikics__s999:298s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:262s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:4625/24564MiB:0%260812-05:05:40 (UTC+8)
1543tasks total=14270 running=3 done=2899 failed=0 blocked=0 pending=11368 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:3897s;cuda2:trace_context:trace_bank__n2__HopQuota-128-b__llama3_8b__wikics__s999:117s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s3407:81s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:4605/24564MiB:96%260812-05:02:39 (UTC+8)
1542tasks total=14270 running=3 done=2897 failed=0 blocked=0 pending=11370 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:3716s;cuda2:trace_context:trace_bank__n2__HopQuota-128-a__llama3_8b__wikics__s999:761s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:2983s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:98%;cuda3:4625/24564MiB:0%260812-04:59:38 (UTC+8)
1541tasks total=14270 running=3 done=2897 failed=0 blocked=0 pending=11370 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:3535s;cuda2:trace_context:trace_bank__n2__HopQuota-128-a__llama3_8b__wikics__s999:580s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:2802s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-04:56:37 (UTC+8)
1540tasks total=14270 running=3 done=2897 failed=0 blocked=0 pending=11370 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:3354s;cuda2:trace_context:trace_bank__n2__HopQuota-128-a__llama3_8b__wikics__s999:399s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:2621s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:4625/24564MiB:0%260812-04:53:36 (UTC+8)
1539tasks total=14270 running=3 done=2897 failed=0 blocked=0 pending=11370 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:3173s;cuda2:trace_context:trace_bank__n2__HopQuota-128-a__llama3_8b__wikics__s999:218s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:2440s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:98%;cuda3:4625/24564MiB:0%260812-04:50:35 (UTC+8)
1538tasks total=14270 running=3 done=2897 failed=0 blocked=0 pending=11370 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:2992s;cuda2:trace_context:trace_bank__n2__HopQuota-128-a__llama3_8b__wikics__s999:37s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:2259s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:4625/24564MiB:0%260812-04:47:34 (UTC+8)
1537tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:2811s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:4328s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:2078s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:4625/24564MiB:0%260812-04:44:33 (UTC+8)
1536tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:2630s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:4148s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:1897s | GPU cuda1:7851/24564MiB:4%;cuda2:7851/24564MiB:96%;cuda3:4625/24564MiB:0%260812-04:41:32 (UTC+8)
1535tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:2449s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:3966s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:1716s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4625/24564MiB:0%260812-04:38:31 (UTC+8)
1534tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:2268s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:3785s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:1535s | GPU cuda1:7851/24564MiB:74%;cuda2:7851/24564MiB:96%;cuda3:4625/24564MiB:0%260812-04:35:30 (UTC+8)
1533tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:2086s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:3604s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:1353s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:4625/24564MiB:97%260812-04:32:29 (UTC+8)
1532tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:1906s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:3423s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:1173s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4625/24564MiB:0%260812-04:29:28 (UTC+8)
1531tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:1725s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:3243s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:992s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:4625/24564MiB:0%260812-04:26:27 (UTC+8)
1530tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:1544s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:3061s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:811s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:4625/24564MiB:0%260812-04:23:26 (UTC+8)
1529tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:1363s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:2881s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:630s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:4625/24564MiB:0%260812-04:20:25 (UTC+8)
1528tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:1181s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:2699s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:448s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:4625/24564MiB:97%260812-04:17:24 (UTC+8)
1527tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:1001s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:2518s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:267s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:4625/24564MiB:0%260812-04:14:23 (UTC+8)
1526tasks total=14270 running=3 done=2895 failed=0 blocked=0 pending=11372 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:820s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:2337s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s3407:87s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:4625/24564MiB:0%260812-04:11:22 (UTC+8)
1525tasks total=14270 running=3 done=2893 failed=0 blocked=0 pending=11374 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:639s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:2157s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:2863s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-04:08:21 (UTC+8)
1524tasks total=14270 running=3 done=2893 failed=0 blocked=0 pending=11374 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:458s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:1975s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:2682s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:97%260812-04:05:20 (UTC+8)
1523tasks total=14270 running=3 done=2893 failed=0 blocked=0 pending=11374 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:277s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:1795s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:2501s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-04:02:19 (UTC+8)
1522tasks total=14270 running=3 done=2893 failed=0 blocked=0 pending=11374 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s3407:96s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:1613s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:2320s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:94%;cuda3:4565/24564MiB:50%260812-03:59:18 (UTC+8)
1521tasks total=14270 running=3 done=2890 failed=0 blocked=0 pending=11377 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:3538s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:1433s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:2139s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-03:56:17 (UTC+8)
1520tasks total=14270 running=3 done=2890 failed=0 blocked=0 pending=11377 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:3357s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:1252s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:1959s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-03:53:16 (UTC+8)
1519tasks total=14270 running=3 done=2890 failed=0 blocked=0 pending=11377 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:3175s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:1070s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:1777s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-03:50:15 (UTC+8)
1518tasks total=14270 running=3 done=2890 failed=0 blocked=0 pending=11377 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:2994s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:889s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:1596s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4565/24564MiB:0%260812-03:47:14 (UTC+8)
1517tasks total=14270 running=3 done=2890 failed=0 blocked=0 pending=11377 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:2813s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:708s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:1415s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-03:44:13 (UTC+8)
1516tasks total=14270 running=3 done=2890 failed=0 blocked=0 pending=11377 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:2632s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:527s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:1234s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:97%260812-03:41:12 (UTC+8)
1515tasks total=14270 running=3 done=2890 failed=0 blocked=0 pending=11377 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:2452s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:346s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:1053s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:4565/24564MiB:0%260812-03:38:11 (UTC+8)
1514tasks total=14270 running=3 done=2890 failed=0 blocked=0 pending=11377 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:2270s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s999:165s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:872s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:4565/24564MiB:0%260812-03:35:10 (UTC+8)
1513tasks total=14270 running=3 done=2889 failed=0 blocked=0 pending=11378 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:2089s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:4458s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:691s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-03:32:09 (UTC+8)
1512tasks total=14270 running=3 done=2889 failed=0 blocked=0 pending=11378 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:1907s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:4276s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:508s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-03:29:08 (UTC+8)
1511tasks total=14270 running=3 done=2889 failed=0 blocked=0 pending=11378 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:1726s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:4095s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:328s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:4565/24564MiB:0%260812-03:26:05 (UTC+8)
1510tasks total=14270 running=3 done=2889 failed=0 blocked=0 pending=11378 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:1545s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:3913s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s3407:146s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:95%260812-03:23:04 (UTC+8)
1509tasks total=14270 running=3 done=2888 failed=0 blocked=0 pending=11379 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:1364s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:3733s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:2872s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-03:20:03 (UTC+8)
1508tasks total=14270 running=3 done=2888 failed=0 blocked=0 pending=11379 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:1183s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:3552s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:2691s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4565/24564MiB:0%260812-03:17:02 (UTC+8)
1507tasks total=14270 running=3 done=2888 failed=0 blocked=0 pending=11379 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:1002s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:3371s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:2510s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:96%260812-03:14:01 (UTC+8)
1506tasks total=14270 running=3 done=2888 failed=0 blocked=0 pending=11379 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:821s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:3190s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:2329s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-03:11:01 (UTC+8)
1505tasks total=14270 running=3 done=2888 failed=0 blocked=0 pending=11379 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:640s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:3009s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:2148s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4565/24564MiB:0%260812-03:08:00 (UTC+8)
1504tasks total=14270 running=3 done=2888 failed=0 blocked=0 pending=11379 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:458s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:2827s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:1966s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:4565/24564MiB:0%260812-03:04:58 (UTC+8)
1503tasks total=14270 running=3 done=2888 failed=0 blocked=0 pending=11379 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:277s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:2646s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:1785s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-03:01:57 (UTC+8)
1502tasks total=14270 running=3 done=2888 failed=0 blocked=0 pending=11379 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s3407:96s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:2465s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:1604s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-02:58:56 (UTC+8)
1501tasks total=14270 running=3 done=2887 failed=0 blocked=0 pending=11380 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:3579s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:2284s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:1423s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:4565/24564MiB:0%260812-02:55:55 (UTC+8)
1500tasks total=14270 running=3 done=2887 failed=0 blocked=0 pending=11380 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:3398s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:2103s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:1242s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:4565/24564MiB:0%260812-02:52:53 (UTC+8)
1499tasks total=14270 running=3 done=2887 failed=0 blocked=0 pending=11380 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:3217s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:1922s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:1061s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-02:49:53 (UTC+8)
1498tasks total=14270 running=3 done=2887 failed=0 blocked=0 pending=11380 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:3036s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:1741s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:880s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-02:46:52 (UTC+8)
1497tasks total=14270 running=3 done=2887 failed=0 blocked=0 pending=11380 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:2855s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:1560s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:699s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:4565/24564MiB:0%260812-02:43:51 (UTC+8)
1496tasks total=14270 running=3 done=2887 failed=0 blocked=0 pending=11380 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:2674s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:1379s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:518s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:4565/24564MiB:0%260812-02:40:50 (UTC+8)
1495tasks total=14270 running=3 done=2887 failed=0 blocked=0 pending=11380 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:2493s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:1198s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:337s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-02:37:49 (UTC+8)
1494tasks total=14270 running=3 done=2887 failed=0 blocked=0 pending=11380 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:2312s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:1017s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s3407:156s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:0%260812-02:34:48 (UTC+8)
1493tasks total=14270 running=3 done=2885 failed=0 blocked=0 pending=11382 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:2131s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:836s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s3407:1547s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:5445/24564MiB:100%260812-02:31:47 (UTC+8)
1492tasks total=14270 running=3 done=2885 failed=0 blocked=0 pending=11382 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:1950s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:655s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s3407:1366s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:5445/24564MiB:94%260812-02:28:46 (UTC+8)
1491tasks total=14270 running=3 done=2885 failed=0 blocked=0 pending=11382 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:1769s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:474s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s3407:1184s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:93%260812-02:25:45 (UTC+8)
1490tasks total=14270 running=3 done=2885 failed=0 blocked=0 pending=11382 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:1588s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:293s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s3407:1003s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:5445/24564MiB:100%260812-02:22:43 (UTC+8)
1489tasks total=14270 running=3 done=2885 failed=0 blocked=0 pending=11382 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:1407s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s999:112s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s3407:823s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:5445/24564MiB:100%260812-02:19:42 (UTC+8)
1488tasks total=14270 running=3 done=2883 failed=0 blocked=0 pending=11384 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:1226s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:3540s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s3407:642s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:94%260812-02:16:42 (UTC+8)
1487tasks total=14270 running=3 done=2883 failed=0 blocked=0 pending=11384 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:1045s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:3359s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s3407:460s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-02:13:41 (UTC+8)
1486tasks total=14270 running=3 done=2883 failed=0 blocked=0 pending=11384 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:864s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:3178s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s3407:280s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:95%260812-02:10:40 (UTC+8)
1485tasks total=14270 running=3 done=2883 failed=0 blocked=0 pending=11384 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:683s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:2998s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s3407:99s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:94%260812-02:07:39 (UTC+8)
1484tasks total=14270 running=3 done=2882 failed=0 blocked=0 pending=11385 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:502s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:2816s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s3407:1486s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-02:04:38 (UTC+8)
1483tasks total=14270 running=3 done=2882 failed=0 blocked=0 pending=11385 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:321s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:2635s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s3407:1305s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:5445/24564MiB:95%260812-02:01:37 (UTC+8)
1482tasks total=14270 running=3 done=2882 failed=0 blocked=0 pending=11385 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s3407:140s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:2455s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s3407:1125s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:95%260812-01:58:36 (UTC+8)
1481tasks total=14270 running=3 done=2879 failed=0 blocked=0 pending=11388 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:3149s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:2273s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s3407:943s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:100%260812-01:55:35 (UTC+8)
1480tasks total=14270 running=3 done=2879 failed=0 blocked=0 pending=11388 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:2968s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:2092s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s3407:762s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-01:52:34 (UTC+8)
1479tasks total=14270 running=3 done=2879 failed=0 blocked=0 pending=11388 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:2787s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:1911s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s3407:581s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:0%260812-01:49:32 (UTC+8)
1478tasks total=14270 running=3 done=2879 failed=0 blocked=0 pending=11388 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:2606s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:1730s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s3407:400s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:100%260812-01:46:31 (UTC+8)
1477tasks total=14270 running=3 done=2879 failed=0 blocked=0 pending=11388 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:2425s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:1549s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s3407:219s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:97%260812-01:43:30 (UTC+8)
1476tasks total=14270 running=3 done=2879 failed=0 blocked=0 pending=11388 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:2244s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:1368s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s3407:38s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-01:40:30 (UTC+8)
1475tasks total=14270 running=3 done=2877 failed=0 blocked=0 pending=11390 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:2063s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:1187s;cuda3:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__wikics__s3407:783s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:4885/24564MiB:96%260812-01:37:29 (UTC+8)
1474tasks total=14270 running=3 done=2877 failed=0 blocked=0 pending=11390 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:1882s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:1006s;cuda3:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__wikics__s3407:602s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:96%260812-01:34:27 (UTC+8)
1473tasks total=14270 running=3 done=2877 failed=0 blocked=0 pending=11390 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:1701s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:825s;cuda3:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__wikics__s3407:422s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:96%260812-01:31:27 (UTC+8)
1472tasks total=14270 running=3 done=2877 failed=0 blocked=0 pending=11390 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:1520s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:645s;cuda3:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__wikics__s3407:241s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:4885/24564MiB:96%260812-01:28:26 (UTC+8)
1471tasks total=14270 running=3 done=2877 failed=0 blocked=0 pending=11390 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:1339s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:463s;cuda3:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__wikics__s3407:60s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:0%260812-01:25:25 (UTC+8)
1470tasks total=14270 running=3 done=2876 failed=0 blocked=0 pending=11391 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:1158s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:283s;cuda3:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__wikics__s3407:802s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:96%260812-01:22:24 (UTC+8)
1469tasks total=14270 running=3 done=2876 failed=0 blocked=0 pending=11391 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:977s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s999:102s;cuda3:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__wikics__s3407:621s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:100%260812-01:19:23 (UTC+8)
1468tasks total=14270 running=3 done=2875 failed=0 blocked=0 pending=11392 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:797s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:3522s;cuda3:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__wikics__s3407:441s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:96%260812-01:16:22 (UTC+8)
1467tasks total=14270 running=3 done=2875 failed=0 blocked=0 pending=11392 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:616s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:3341s;cuda3:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__wikics__s3407:260s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:96%260812-01:13:21 (UTC+8)
1466tasks total=14270 running=3 done=2875 failed=0 blocked=0 pending=11392 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:435s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:3160s;cuda3:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__wikics__s3407:79s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:96%260812-01:10:20 (UTC+8)
1465tasks total=14270 running=3 done=2873 failed=0 blocked=0 pending=11394 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:254s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:2979s;cuda3:trace_context:trace_bank__n1__Semantic-128-b__llama31_4b__wikics__s3407:427s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4565/24564MiB:97%260812-01:07:19 (UTC+8)
1464tasks total=14270 running=3 done=2873 failed=0 blocked=0 pending=11394 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s3407:73s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:2798s;cuda3:trace_context:trace_bank__n1__Semantic-128-b__llama31_4b__wikics__s3407:246s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:95%260812-01:04:18 (UTC+8)
1463tasks total=14270 running=3 done=2872 failed=0 blocked=0 pending=11395 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:3102s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:2617s;cuda3:trace_context:trace_bank__n1__Semantic-128-b__llama31_4b__wikics__s3407:65s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4565/24564MiB:97%260812-01:01:17 (UTC+8)
1462tasks total=14270 running=3 done=2871 failed=0 blocked=0 pending=11396 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:2921s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:2436s;cuda3:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__wikics__s3407:411s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:97%260812-00:58:16 (UTC+8)
1461tasks total=14270 running=3 done=2871 failed=0 blocked=0 pending=11396 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:2740s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:2255s;cuda3:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__wikics__s3407:230s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4565/24564MiB:95%260812-00:55:15 (UTC+8)
1460tasks total=14270 running=3 done=2871 failed=0 blocked=0 pending=11396 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:2559s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:2075s;cuda3:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__wikics__s3407:49s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4565/24564MiB:97%260812-00:52:15 (UTC+8)
1459tasks total=14270 running=3 done=2869 failed=0 blocked=0 pending=11398 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:2377s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:1892s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s3407:1412s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-00:49:14 (UTC+8)
1458tasks total=14270 running=3 done=2869 failed=0 blocked=0 pending=11398 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:2196s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:1711s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s3407:1231s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-00:46:11 (UTC+8)
1457tasks total=14270 running=3 done=2869 failed=0 blocked=0 pending=11398 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:2014s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:1529s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s3407:1048s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:95%260812-00:43:10 (UTC+8)
1456tasks total=14270 running=3 done=2869 failed=0 blocked=0 pending=11398 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:1833s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:1348s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s3407:867s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:96%260812-00:40:08 (UTC+8)
1455tasks total=14270 running=3 done=2869 failed=0 blocked=0 pending=11398 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:1651s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:1167s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s3407:686s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-00:37:07 (UTC+8)
1454tasks total=14270 running=3 done=2869 failed=0 blocked=0 pending=11398 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:1471s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:986s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s3407:505s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:5445/24564MiB:100%260812-00:34:06 (UTC+8)
1453tasks total=14270 running=3 done=2869 failed=0 blocked=0 pending=11398 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:1289s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:804s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s3407:324s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:100%260812-00:31:05 (UTC+8)
1452tasks total=14270 running=3 done=2869 failed=0 blocked=0 pending=11398 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:1108s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:623s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s3407:143s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-00:28:04 (UTC+8)
1451tasks total=14270 running=3 done=2868 failed=0 blocked=0 pending=11399 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:927s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:442s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s3407:1504s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:100%260812-00:25:02 (UTC+8)
1450tasks total=14270 running=3 done=2868 failed=0 blocked=0 pending=11399 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:746s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:261s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s3407:1323s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-00:22:02 (UTC+8)
1449tasks total=14270 running=3 done=2868 failed=0 blocked=0 pending=11399 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:566s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s999:81s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s3407:1142s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:94%260812-00:19:01 (UTC+8)
1448tasks total=14270 running=3 done=2866 failed=0 blocked=0 pending=11401 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:383s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:3113s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s3407:960s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:5445/24564MiB:95%260812-00:16:00 (UTC+8)
1447tasks total=14270 running=3 done=2866 failed=0 blocked=0 pending=11401 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:202s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:2932s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s3407:779s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-00:12:58 (UTC+8)
1446tasks total=14270 running=3 done=2866 failed=0 blocked=0 pending=11401 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s3407:22s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:2751s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s3407:598s | GPU cuda1:7731/24564MiB:24%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-00:09:57 (UTC+8)
1445tasks total=14270 running=3 done=2863 failed=0 blocked=0 pending=11404 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:2845s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:2571s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s3407:417s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-00:06:56 (UTC+8)
1444tasks total=14270 running=3 done=2863 failed=0 blocked=0 pending=11404 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:2664s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:2389s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s3407:236s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:95%260812-00:03:55 (UTC+8)
1443tasks total=14270 running=3 done=2863 failed=0 blocked=0 pending=11404 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:2483s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:2208s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s3407:55s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:5445/24564MiB:100%260812-00:00:54 (UTC+8)
1442tasks total=14270 running=3 done=2861 failed=0 blocked=0 pending=11406 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:2302s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:2027s;cuda3:trace_context:trace_bank__n1__Random-512-b__llama31_4b__wikics__s3407:775s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:96%260811-23:57:53 (UTC+8)
1441tasks total=14270 running=3 done=2861 failed=0 blocked=0 pending=11406 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:2121s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:1846s;cuda3:trace_context:trace_bank__n1__Random-512-b__llama31_4b__wikics__s3407:594s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:96%260811-23:54:52 (UTC+8)
1440tasks total=14270 running=3 done=2861 failed=0 blocked=0 pending=11406 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:1940s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:1665s;cuda3:trace_context:trace_bank__n1__Random-512-b__llama31_4b__wikics__s3407:413s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:96%260811-23:51:51 (UTC+8)
1439tasks total=14270 running=3 done=2861 failed=0 blocked=0 pending=11406 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:1758s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:1483s;cuda3:trace_context:trace_bank__n1__Random-512-b__llama31_4b__wikics__s3407:231s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:96%260811-23:48:49 (UTC+8)
1438tasks total=14270 running=3 done=2861 failed=0 blocked=0 pending=11406 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:1577s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:1302s;cuda3:trace_context:trace_bank__n1__Random-512-b__llama31_4b__wikics__s3407:50s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:4885/24564MiB:96%260811-23:45:47 (UTC+8)
1437tasks total=14270 running=3 done=2860 failed=0 blocked=0 pending=11407 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:1396s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:1121s;cuda3:trace_context:trace_bank__n1__Random-512-a__llama31_4b__wikics__s3407:768s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:100%260811-23:42:47 (UTC+8)
1436tasks total=14270 running=3 done=2860 failed=0 blocked=0 pending=11407 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:1214s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:939s;cuda3:trace_context:trace_bank__n1__Random-512-a__llama31_4b__wikics__s3407:586s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:96%260811-23:39:46 (UTC+8)
1435tasks total=14270 running=3 done=2860 failed=0 blocked=0 pending=11407 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:1033s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:758s;cuda3:trace_context:trace_bank__n1__Random-512-a__llama31_4b__wikics__s3407:405s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:4885/24564MiB:0%260811-23:36:44 (UTC+8)
1434tasks total=14270 running=3 done=2860 failed=0 blocked=0 pending=11407 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:853s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:578s;cuda3:trace_context:trace_bank__n1__Random-512-a__llama31_4b__wikics__s3407:224s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:96%260811-23:33:43 (UTC+8)
1433tasks total=14270 running=3 done=2860 failed=0 blocked=0 pending=11407 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:670s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:395s;cuda3:trace_context:trace_bank__n1__Random-512-a__llama31_4b__wikics__s3407:42s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:4885/24564MiB:96%260811-23:30:42 (UTC+8)
1432tasks total=14270 running=3 done=2858 failed=0 blocked=0 pending=11409 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:489s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:214s;cuda3:trace_context:trace_bank__n1__Random-256-b__llama31_4b__wikics__s3407:741s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260811-23:27:40 (UTC+8)
1431tasks total=14270 running=3 done=2858 failed=0 blocked=0 pending=11409 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:308s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s999:33s;cuda3:trace_context:trace_bank__n1__Random-256-b__llama31_4b__wikics__s3407:561s | GPU cuda1:7851/24564MiB:95%;cuda2:7835/24564MiB:0%;cuda3:6529/24564MiB:100%260811-23:24:39 (UTC+8)
1430tasks total=14270 running=3 done=2857 failed=0 blocked=0 pending=11410 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s3407:127s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:3029s;cuda3:trace_context:trace_bank__n1__Random-256-b__llama31_4b__wikics__s3407:380s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260811-23:21:38 (UTC+8)
1429tasks total=14270 running=3 done=2856 failed=0 blocked=0 pending=11411 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:2930s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:2848s;cuda3:trace_context:trace_bank__n1__Random-256-b__llama31_4b__wikics__s3407:199s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:99%260811-23:18:37 (UTC+8)
1428tasks total=14270 running=3 done=2856 failed=0 blocked=0 pending=11411 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:2749s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:2667s;cuda3:trace_context:trace_bank__n1__Random-256-b__llama31_4b__wikics__s3407:18s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:6463/24564MiB:48%260811-23:15:36 (UTC+8)
1427tasks total=14270 running=3 done=2855 failed=0 blocked=0 pending=11412 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:2568s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:2486s;cuda3:trace_context:trace_bank__n1__Random-256-a__llama31_4b__wikics__s3407:767s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:99%260811-23:12:35 (UTC+8)
1426tasks total=14270 running=3 done=2855 failed=0 blocked=0 pending=11412 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:2387s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:2305s;cuda3:trace_context:trace_bank__n1__Random-256-a__llama31_4b__wikics__s3407:586s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260811-23:09:34 (UTC+8)
1425tasks total=14270 running=3 done=2855 failed=0 blocked=0 pending=11412 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:2206s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:2124s;cuda3:trace_context:trace_bank__n1__Random-256-a__llama31_4b__wikics__s3407:405s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-23:06:33 (UTC+8)
1424tasks total=14270 running=3 done=2855 failed=0 blocked=0 pending=11412 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:2025s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:1943s;cuda3:trace_context:trace_bank__n1__Random-256-a__llama31_4b__wikics__s3407:224s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:99%260811-23:03:32 (UTC+8)
1423tasks total=14270 running=3 done=2855 failed=0 blocked=0 pending=11412 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:1843s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:1761s;cuda3:trace_context:trace_bank__n1__Random-256-a__llama31_4b__wikics__s3407:42s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:100%260811-23:00:31 (UTC+8)
1422tasks total=14270 running=3 done=2853 failed=0 blocked=0 pending=11414 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:1661s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:1579s;cuda3:trace_context:trace_bank__n1__Random-128-b__llama31_4b__wikics__s3407:596s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:99%260811-22:57:29 (UTC+8)
1421tasks total=14270 running=3 done=2853 failed=0 blocked=0 pending=11414 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:1479s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:1397s;cuda3:trace_context:trace_bank__n1__Random-128-b__llama31_4b__wikics__s3407:414s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:99%260811-22:54:27 (UTC+8)
1420tasks total=14270 running=3 done=2853 failed=0 blocked=0 pending=11414 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:1298s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:1216s;cuda3:trace_context:trace_bank__n1__Random-128-b__llama31_4b__wikics__s3407:233s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:99%260811-22:51:25 (UTC+8)
1419tasks total=14270 running=3 done=2853 failed=0 blocked=0 pending=11414 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:1117s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:1035s;cuda3:trace_context:trace_bank__n1__Random-128-b__llama31_4b__wikics__s3407:53s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:98%260811-22:48:24 (UTC+8)
1418tasks total=14270 running=3 done=2852 failed=0 blocked=0 pending=11415 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:930s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:848s;cuda3:trace_context:trace_bank__n1__Random-128-a__llama31_4b__wikics__s3407:599s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:34%;cuda3:6469/24564MiB:98%260811-22:45:23 (UTC+8)
1417tasks total=14270 running=3 done=2852 failed=0 blocked=0 pending=11415 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:748s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:666s;cuda3:trace_context:trace_bank__n1__Random-128-a__llama31_4b__wikics__s3407:418s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:98%260811-22:42:16 (UTC+8)
1416tasks total=14270 running=3 done=2852 failed=0 blocked=0 pending=11415 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:568s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:486s;cuda3:trace_context:trace_bank__n1__Random-128-a__llama31_4b__wikics__s3407:237s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:100%260811-22:39:15 (UTC+8)
1415tasks total=14270 running=3 done=2852 failed=0 blocked=0 pending=11415 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:387s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:305s;cuda3:trace_context:trace_bank__n1__Random-128-a__llama31_4b__wikics__s3407:56s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:98%260811-22:36:14 (UTC+8)
1414tasks total=14270 running=3 done=2850 failed=0 blocked=0 pending=11417 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:206s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s999:124s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:2176s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7369/24564MiB:100%260811-22:33:13 (UTC+8)
1413tasks total=14270 running=3 done=2848 failed=0 blocked=0 pending=11419 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s3407:23s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:2932s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:1994s | GPU cuda1:7835/24564MiB:0%;cuda2:7851/24564MiB:98%;cuda3:7369/24564MiB:100%260811-22:30:12 (UTC+8)
1412tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:2194s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:2751s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:1813s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7369/24564MiB:100%260811-22:27:09 (UTC+8)
1411tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:2013s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:2570s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:1632s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:7369/24564MiB:100%260811-22:24:09 (UTC+8)
1410tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:1833s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:2390s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:1451s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7369/24564MiB:100%260811-22:21:08 (UTC+8)
1409tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:1652s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:2209s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:1270s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:7369/24564MiB:100%260811-22:18:07 (UTC+8)
1408tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:1469s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:2026s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:1088s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7369/24564MiB:98%260811-22:15:06 (UTC+8)
1407tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:1288s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:1845s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:907s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:7369/24564MiB:100%260811-22:12:04 (UTC+8)
1406tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:1105s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:1662s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:723s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:7369/24564MiB:100%260811-22:09:03 (UTC+8)
1405tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:924s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:1481s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:542s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-22:05:59 (UTC+8)
1404tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:743s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:1300s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:361s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-22:02:58 (UTC+8)
1403tasks total=14270 running=3 done=2845 failed=0 blocked=0 pending=11422 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:562s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:1119s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s3407:180s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:100%260811-21:59:57 (UTC+8)
1402tasks total=14270 running=3 done=2844 failed=0 blocked=0 pending=11423 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:381s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:938s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:2299s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:4367/24564MiB:45%260811-21:56:56 (UTC+8)
1401tasks total=14270 running=3 done=2844 failed=0 blocked=0 pending=11423 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:200s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:757s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:2119s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7369/24564MiB:97%260811-21:53:55 (UTC+8)
1400tasks total=14270 running=3 done=2844 failed=0 blocked=0 pending=11423 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s3407:19s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:576s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:1938s | GPU cuda1:7713/24564MiB:24%;cuda2:7851/24564MiB:0%;cuda3:7369/24564MiB:100%260811-21:50:54 (UTC+8)
1399tasks total=14270 running=3 done=2843 failed=0 blocked=0 pending=11424 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:2185s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:395s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:1756s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7369/24564MiB:100%260811-21:47:53 (UTC+8)
1398tasks total=14270 running=3 done=2843 failed=0 blocked=0 pending=11424 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:2004s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:213s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:1575s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:98%;cuda3:7369/24564MiB:100%260811-21:44:52 (UTC+8)
1397tasks total=14270 running=3 done=2843 failed=0 blocked=0 pending=11424 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:1822s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s999:32s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:1393s | GPU cuda1:7851/24564MiB:100%;cuda2:7835/24564MiB:0%;cuda3:7369/24564MiB:47%260811-21:41:51 (UTC+8)
1396tasks total=14270 running=3 done=2842 failed=0 blocked=0 pending=11425 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:1642s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:2893s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:1213s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7369/24564MiB:97%260811-21:38:49 (UTC+8)
1395tasks total=14270 running=3 done=2842 failed=0 blocked=0 pending=11425 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:1461s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:2712s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:1032s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7369/24564MiB:100%260811-21:35:48 (UTC+8)
1394tasks total=14270 running=3 done=2842 failed=0 blocked=0 pending=11425 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:1280s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:2531s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:851s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7369/24564MiB:100%260811-21:32:48 (UTC+8)
1393tasks total=14270 running=3 done=2842 failed=0 blocked=0 pending=11425 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:1099s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:2350s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:670s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7369/24564MiB:100%260811-21:29:47 (UTC+8)
1392tasks total=14270 running=3 done=2842 failed=0 blocked=0 pending=11425 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:918s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:2170s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:489s | GPU cuda1:7851/24564MiB:99%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:96%260811-21:26:46 (UTC+8)
1391tasks total=14270 running=3 done=2842 failed=0 blocked=0 pending=11425 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:736s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:1988s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:307s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:96%260811-21:23:45 (UTC+8)
1390tasks total=14270 running=3 done=2842 failed=0 blocked=0 pending=11425 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:555s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:1807s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s3407:126s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-21:20:43 (UTC+8)
1389tasks total=14270 running=3 done=2840 failed=0 blocked=0 pending=11427 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:374s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:1626s;cuda3:trace_context:trace_bank__n1__Full-256-b__llama31_4b__wikics__s3407:884s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:98%260811-21:17:42 (UTC+8)
1388tasks total=14270 running=3 done=2840 failed=0 blocked=0 pending=11427 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:194s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:1445s;cuda3:trace_context:trace_bank__n1__Full-256-b__llama31_4b__wikics__s3407:703s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-21:14:41 (UTC+8)
1387tasks total=14270 running=3 done=2840 failed=0 blocked=0 pending=11427 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s3407:11s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:1263s;cuda3:trace_context:trace_bank__n1__Full-256-b__llama31_4b__wikics__s3407:521s | GPU cuda1:7659/24564MiB:23%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:91%260811-21:11:40 (UTC+8)
1386tasks total=14270 running=3 done=2837 failed=0 blocked=0 pending=11430 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s3407:1314s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:1080s;cuda3:trace_context:trace_bank__n1__Full-256-b__llama31_4b__wikics__s3407:338s | GPU cuda1:7851/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260811-21:08:38 (UTC+8)
1385tasks total=14270 running=3 done=2837 failed=0 blocked=0 pending=11430 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s3407:1134s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:899s;cuda3:trace_context:trace_bank__n1__Full-256-b__llama31_4b__wikics__s3407:157s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:99%260811-21:05:35 (UTC+8)
1384tasks total=14270 running=3 done=2836 failed=0 blocked=0 pending=11431 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s3407:953s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:718s;cuda3:trace_context:trace_bank__n1__Full-256-a__llama31_4b__wikics__s3407:911s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-21:02:35 (UTC+8)
1383tasks total=14270 running=3 done=2836 failed=0 blocked=0 pending=11431 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s3407:770s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:536s;cuda3:trace_context:trace_bank__n1__Full-256-a__llama31_4b__wikics__s3407:729s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:99%260811-20:59:34 (UTC+8)
1382tasks total=14270 running=3 done=2836 failed=0 blocked=0 pending=11431 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s3407:589s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:355s;cuda3:trace_context:trace_bank__n1__Full-256-a__llama31_4b__wikics__s3407:548s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-20:56:31 (UTC+8)
1381tasks total=14270 running=3 done=2836 failed=0 blocked=0 pending=11431 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s3407:408s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s999:174s;cuda3:trace_context:trace_bank__n1__Full-256-a__llama31_4b__wikics__s3407:367s | GPU cuda1:7851/24564MiB:27%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-20:53:30 (UTC+8)
1380tasks total=14270 running=3 done=2834 failed=0 blocked=0 pending=11433 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s3407:228s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:2476s;cuda3:trace_context:trace_bank__n1__Full-256-a__llama31_4b__wikics__s3407:186s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:6529/24564MiB:98%260811-20:50:29 (UTC+8)
1379tasks total=14270 running=3 done=2834 failed=0 blocked=0 pending=11433 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s3407:46s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:2294s;cuda3:trace_context:trace_bank__n1__Full-256-a__llama31_4b__wikics__s3407:5s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:2305/24564MiB:46%260811-20:47:29 (UTC+8)
1378tasks total=14270 running=3 done=2831 failed=0 blocked=0 pending=11436 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s3407:1347s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:2113s;cuda3:trace_context:trace_bank__n1__Full-128-b__llama31_4b__wikics__s3407:555s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:99%260811-20:44:27 (UTC+8)
1377tasks total=14270 running=3 done=2831 failed=0 blocked=0 pending=11436 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s3407:1166s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:1932s;cuda3:trace_context:trace_bank__n1__Full-128-b__llama31_4b__wikics__s3407:374s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:94%;cuda3:6469/24564MiB:99%260811-20:41:26 (UTC+8)
1376tasks total=14270 running=3 done=2831 failed=0 blocked=0 pending=11436 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s3407:985s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:1751s;cuda3:trace_context:trace_bank__n1__Full-128-b__llama31_4b__wikics__s3407:194s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:98%260811-20:38:25 (UTC+8)
1375tasks total=14270 running=3 done=2831 failed=0 blocked=0 pending=11436 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s3407:804s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:1570s;cuda3:trace_context:trace_bank__n1__Full-128-b__llama31_4b__wikics__s3407:12s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:6347/24564MiB:66%260811-20:35:24 (UTC+8)
1374tasks total=14270 running=3 done=2830 failed=0 blocked=0 pending=11437 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s3407:622s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:1388s;cuda3:trace_context:trace_bank__n1__Full-128-a__llama31_4b__wikics__s3407:561s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:99%260811-20:32:23 (UTC+8)
1373tasks total=14270 running=3 done=2830 failed=0 blocked=0 pending=11437 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s3407:441s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:1207s;cuda3:trace_context:trace_bank__n1__Full-128-a__llama31_4b__wikics__s3407:380s | GPU cuda1:7851/24564MiB:46%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:98%260811-20:29:21 (UTC+8)
1372tasks total=14270 running=3 done=2830 failed=0 blocked=0 pending=11437 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s3407:260s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:1026s;cuda3:trace_context:trace_bank__n1__Full-128-a__llama31_4b__wikics__s3407:199s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:100%260811-20:26:20 (UTC+8)
1371tasks total=14270 running=3 done=2830 failed=0 blocked=0 pending=11437 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s3407:79s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:845s;cuda3:trace_context:trace_bank__n1__Full-128-a__llama31_4b__wikics__s3407:18s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:6469/24564MiB:88%260811-20:23:19 (UTC+8)
1370tasks total=14270 running=3 done=2825 failed=0 blocked=0 pending=11442 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-b__llama3_8b__wikics__s3407:800s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:664s;cuda3:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__wikics__s3407:796s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:99%260811-20:20:18 (UTC+8)
1369tasks total=14270 running=3 done=2825 failed=0 blocked=0 pending=11442 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-b__llama3_8b__wikics__s3407:617s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:481s;cuda3:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__wikics__s3407:613s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:98%260811-20:17:17 (UTC+8)
1368tasks total=14270 running=3 done=2825 failed=0 blocked=0 pending=11442 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-b__llama3_8b__wikics__s3407:436s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:301s;cuda3:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__wikics__s3407:432s | GPU cuda1:7851/24564MiB:75%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:98%260811-20:14:15 (UTC+8)
1367tasks total=14270 running=3 done=2825 failed=0 blocked=0 pending=11442 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-b__llama3_8b__wikics__s3407:255s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s999:120s;cuda3:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__wikics__s3407:251s | GPU cuda1:7851/24564MiB:39%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:99%260811-20:11:14 (UTC+8)
1366tasks total=14270 running=3 done=2824 failed=0 blocked=0 pending=11443 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-b__llama3_8b__wikics__s3407:74s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:2413s;cuda3:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__wikics__s3407:70s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:98%260811-20:08:13 (UTC+8)
1365tasks total=14270 running=3 done=2822 failed=0 blocked=0 pending=11445 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s3407:792s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:2232s;cuda3:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__wikics__s3407:847s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:100%260811-20:05:12 (UTC+8)
1364tasks total=14270 running=3 done=2822 failed=0 blocked=0 pending=11445 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s3407:611s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:2051s;cuda3:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__wikics__s3407:666s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:97%260811-20:02:11 (UTC+8)
1363tasks total=14270 running=3 done=2822 failed=0 blocked=0 pending=11445 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s3407:429s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:1869s;cuda3:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__wikics__s3407:484s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:100%260811-19:59:10 (UTC+8)
1362tasks total=14270 running=3 done=2822 failed=0 blocked=0 pending=11445 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s3407:248s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:1688s;cuda3:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__wikics__s3407:303s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:98%260811-19:56:08 (UTC+8)
1361tasks total=14270 running=3 done=2822 failed=0 blocked=0 pending=11445 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s3407:67s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:1507s;cuda3:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__wikics__s3407:122s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:99%260811-19:53:07 (UTC+8)
1360tasks total=14270 running=3 done=2817 failed=0 blocked=0 pending=11450 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:2211s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:1326s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama31_4b__wikics__s3407:1268s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:100%260811-19:50:06 (UTC+8)
1359tasks total=14270 running=3 done=2817 failed=0 blocked=0 pending=11450 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:2029s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:1144s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama31_4b__wikics__s3407:1086s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:100%260811-19:47:05 (UTC+8)
1358tasks total=14270 running=3 done=2817 failed=0 blocked=0 pending=11450 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:1848s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:963s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama31_4b__wikics__s3407:905s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:52%260811-19:44:03 (UTC+8)
1357tasks total=14270 running=3 done=2817 failed=0 blocked=0 pending=11450 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:1667s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:782s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama31_4b__wikics__s3407:724s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:100%260811-19:41:02 (UTC+8)
1356tasks total=14270 running=3 done=2817 failed=0 blocked=0 pending=11450 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:1486s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:601s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama31_4b__wikics__s3407:543s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:97%260811-19:38:01 (UTC+8)
1355tasks total=14270 running=3 done=2817 failed=0 blocked=0 pending=11450 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:1305s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:420s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama31_4b__wikics__s3407:362s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:100%260811-19:35:00 (UTC+8)
1354tasks total=14270 running=3 done=2817 failed=0 blocked=0 pending=11450 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:1124s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:239s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama31_4b__wikics__s3407:181s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:63%;cuda3:6789/24564MiB:100%260811-19:31:59 (UTC+8)
1353tasks total=14270 running=3 done=2816 failed=0 blocked=0 pending=11451 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:943s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s999:58s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s3407:1324s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:95%;cuda3:1917/24564MiB:56%260811-19:28:58 (UTC+8)
1352tasks total=14270 running=3 done=2814 failed=0 blocked=0 pending=11453 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:762s;cuda2:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s999:1441s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s3407:1143s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:99%260811-19:25:57 (UTC+8)
1351tasks total=14270 running=3 done=2814 failed=0 blocked=0 pending=11453 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:582s;cuda2:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s999:1260s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s3407:962s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:100%260811-19:22:57 (UTC+8)
1350tasks total=14270 running=3 done=2814 failed=0 blocked=0 pending=11453 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:401s;cuda2:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s999:1079s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s3407:781s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:99%260811-19:19:56 (UTC+8)
1349tasks total=14270 running=3 done=2814 failed=0 blocked=0 pending=11453 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:217s;cuda2:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s999:896s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s3407:598s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:100%260811-19:16:55 (UTC+8)
1348tasks total=14270 running=3 done=2814 failed=0 blocked=0 pending=11453 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s3407:36s;cuda2:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s999:715s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s3407:417s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:100%260811-19:13:52 (UTC+8)
1347tasks total=14270 running=3 done=2813 failed=0 blocked=0 pending=11454 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:2175s;cuda2:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s999:533s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s3407:235s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:100%260811-19:10:50 (UTC+8)
1346tasks total=14270 running=3 done=2813 failed=0 blocked=0 pending=11454 | results done=960/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:1994s;cuda2:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s999:351s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s3407:53s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:100%260811-19:07:48 (UTC+8)
1345tasks total=14270 running=3 done=2811 failed=0 blocked=0 pending=11456 | results done=959/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:1813s;cuda2:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s999:171s;cuda3:g4_teacher:g4_teacher__llama31_4b__wikics__s3407:579s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:99%260811-19:04:47 (UTC+8)
1344tasks total=14270 running=3 done=2810 failed=0 blocked=0 pending=11457 | results done=959/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:1632s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s999:1541s;cuda3:g4_teacher:g4_teacher__llama31_4b__wikics__s3407:398s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:100%260811-19:01:46 (UTC+8)
1343tasks total=14270 running=3 done=2810 failed=0 blocked=0 pending=11457 | results done=959/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:1451s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s999:1360s;cuda3:g4_teacher:g4_teacher__llama31_4b__wikics__s3407:217s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:98%260811-18:58:45 (UTC+8)
1342tasks total=14270 running=3 done=2810 failed=0 blocked=0 pending=11457 | results done=959/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:1270s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s999:1179s;cuda3:g4_teacher:g4_teacher__llama31_4b__wikics__s3407:35s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:6469/24564MiB:100%260811-18:55:44 (UTC+8)
1341tasks total=14270 running=3 done=2806 failed=0 blocked=0 pending=11461 | results done=956/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:1089s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s999:998s;cuda3:trace_downstream:run__N4_TokenMatchedRandom__llama31_4b__wikics__s3407__gcn3:23s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:98%;cuda3:3541/24564MiB:99%260811-18:52:43 (UTC+8)
1340tasks total=14270 running=3 done=2802 failed=0 blocked=0 pending=11465 | results done=952/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:908s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s999:817s;cuda3:trace_downstream:run__O1_OrderA__llama31_4b__wikics__s3407__gcn3:276s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:3541/24564MiB:99%260811-18:49:42 (UTC+8)
1339tasks total=14270 running=3 done=2802 failed=0 blocked=0 pending=11465 | results done=952/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:727s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s999:636s;cuda3:trace_downstream:run__O1_OrderA__llama31_4b__wikics__s3407__gcn3:95s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:3541/24564MiB:99%260811-18:46:41 (UTC+8)
1338tasks total=14270 running=3 done=2799 failed=0 blocked=0 pending=11468 | results done=949/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:547s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s999:455s;cuda3:trace_downstream:run__I2_LayerwiseKL__llama31_4b__wikics__s3407__gcn3:39s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:3541/24564MiB:99%260811-18:43:40 (UTC+8)
1337tasks total=14270 running=3 done=2796 failed=0 blocked=0 pending=11471 | results done=946/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:366s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s999:275s;cuda3:trace_downstream:run__C6_BudgetSweep__budget_512__llama31_4b__wikics__s3407__gcn3:18s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:3541/24564MiB:99%260811-18:40:39 (UTC+8)
1336tasks total=14270 running=2 done=2793 failed=0 blocked=0 pending=11475 | results done=943/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:185s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s999:93s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:1917/24564MiB:55%260811-18:37:39 (UTC+8)
1335tasks total=14270 running=3 done=2787 failed=0 blocked=0 pending=11480 | results done=939/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s3407:4s;cuda2:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s999:849s;cuda3:trace_downstream:run__C5_LeaveFamilyOut__without_Full__llama31_4b__wikics__s3407__gcn3:17s | GPU cuda1:401/24564MiB:0%;cuda2:7851/24564MiB:8%;cuda3:3541/24564MiB:99%260811-18:34:37 (UTC+8)
1334tasks total=14270 running=3 done=2780 failed=0 blocked=0 pending=11487 | results done=935/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s3407:1282s;cuda2:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s999:669s;cuda3:trace_downstream:run__C2_Semantic__llama31_4b__wikics__s3407__gcn3:41s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:3539/24564MiB:99%260811-18:31:36 (UTC+8)
1333tasks total=14270 running=3 done=2777 failed=0 blocked=0 pending=11490 | results done=932/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s3407:1099s;cuda2:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s999:486s;cuda3:trace_downstream:run__A7_SoftVsHard__soft__llama31_4b__wikics__s3407__gcn3:19s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:3539/24564MiB:99%260811-18:28:36 (UTC+8)
1332tasks total=14270 running=3 done=2774 failed=0 blocked=0 pending=11493 | results done=929/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s3407:918s;cuda2:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s999:305s;cuda3:trace_downstream:run__A4_NoTokenCost__llama31_4b__wikics__s3407__gcn3:2s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:2305/24564MiB:45%260811-18:25:33 (UTC+8)
1331tasks total=14270 running=3 done=2770 failed=0 blocked=0 pending=11497 | results done=925/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s3407:737s;cuda2:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s999:124s;cuda3:trace_downstream:run__T9_Oracle__llama31_4b__wikics__s3407__gcn3:12s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:3539/24564MiB:99%260811-18:22:32 (UTC+8)
1330tasks total=14270 running=3 done=2764 failed=0 blocked=0 pending=11503 | results done=920/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s3407:557s;cuda2:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s999:878s;cuda3:trace_downstream:run__T8_TRACEBilevelHard__llama31_4b__wikics__s3407__gcn2:29s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:3537/24564MiB:99%260811-18:19:31 (UTC+8)
1329tasks total=14270 running=3 done=2759 failed=0 blocked=0 pending=11508 | results done=915/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s3407:375s;cuda2:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s999:697s;cuda3:trace_downstream:run__T7_TRACEBilevelSoft__llama31_4b__wikics__s3407__gcn3:22s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:3541/24564MiB:99%260811-18:16:31 (UTC+8)
1328tasks total=14270 running=3 done=2753 failed=0 blocked=0 pending=11514 | results done=909/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s3407:195s;cuda2:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s999:516s;cuda3:trace_downstream:run__T6_TRACEProxy__llama31_4b__wikics__s3407__gcn3:53s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:3541/24564MiB:99%260811-18:13:29 (UTC+8)
1327tasks total=14270 running=3 done=2749 failed=0 blocked=0 pending=11518 | results done=905/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s3407:13s;cuda2:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s999:334s;cuda3:trace_downstream:run__T5_RealizabilityOnly__llama31_4b__wikics__s3407__sage2:9s | GPU cuda1:7667/24564MiB:19%;cuda2:7851/24564MiB:98%;cuda3:3339/24564MiB:99%260811-18:10:29 (UTC+8)
1326tasks total=14270 running=3 done=2743 failed=0 blocked=0 pending=11524 | results done=900/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s3407:1287s;cuda2:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s999:153s;cuda3:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__wikics__s3407__sage3:183s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:3345/24564MiB:79%260811-18:07:27 (UTC+8)
1325tasks total=14270 running=3 done=2741 failed=0 blocked=0 pending=11526 | results done=900/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s3407:1106s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:2419s;cuda3:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__wikics__s3407__sage3:2s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:2305/24564MiB:46%260811-18:04:26 (UTC+8)
1324tasks total=14270 running=3 done=2739 failed=0 blocked=0 pending=11528 | results done=898/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s3407:924s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:2236s;cuda3:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__wikics__s3407__sage1:79s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:2955/24564MiB:98%260811-18:01:25 (UTC+8)
1323tasks total=14270 running=3 done=2738 failed=0 blocked=0 pending=11529 | results done=897/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s3407:743s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:2055s;cuda3:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__wikics__s3407__gcn3:227s | GPU cuda1:7851/24564MiB:99%;cuda2:7851/24564MiB:100%;cuda3:3541/24564MiB:99%260811-17:58:23 (UTC+8)
1322tasks total=14270 running=3 done=2738 failed=0 blocked=0 pending=11529 | results done=897/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s3407:562s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:1875s;cuda3:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__wikics__s3407__gcn3:46s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:3541/24564MiB:99%260811-17:55:22 (UTC+8)
1321tasks total=14270 running=3 done=2737 failed=0 blocked=0 pending=11530 | results done=896/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s3407:381s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:1694s;cuda3:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__wikics__s3407__gcn2:95s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:100%;cuda3:3457/24564MiB:99%260811-17:52:21 (UTC+8)
1320tasks total=14270 running=3 done=2736 failed=0 blocked=0 pending=11531 | results done=895/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s3407:200s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:1513s;cuda3:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__wikics__s3407__gcn1:53s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:3413/24564MiB:99%260811-17:49:20 (UTC+8)
1319tasks total=14270 running=3 done=2732 failed=0 blocked=0 pending=11535 | results done=891/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s3407:19s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:1332s;cuda3:trace_downstream:run__T3_TeacherOptGlobal__llama31_4b__wikics__s3407__gcn3:13s | GPU cuda1:7715/24564MiB:24%;cuda2:7851/24564MiB:100%;cuda3:3541/24564MiB:99%260811-17:46:19 (UTC+8)
1318tasks total=14270 running=3 done=2723 failed=0 blocked=0 pending=11544 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-b__llama3_8b__wikics__s3407:891s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:1151s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:2113s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:96%260811-17:43:18 (UTC+8)
1317tasks total=14270 running=3 done=2723 failed=0 blocked=0 pending=11544 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-b__llama3_8b__wikics__s3407:710s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:970s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:1932s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-17:40:17 (UTC+8)
1316tasks total=14270 running=3 done=2723 failed=0 blocked=0 pending=11544 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-b__llama3_8b__wikics__s3407:529s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:789s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:1752s | GPU cuda1:7851/24564MiB:63%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-17:37:17 (UTC+8)
1315tasks total=14270 running=3 done=2723 failed=0 blocked=0 pending=11544 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-b__llama3_8b__wikics__s3407:348s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:609s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:1571s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:100%260811-17:34:16 (UTC+8)
1314tasks total=14270 running=3 done=2723 failed=0 blocked=0 pending=11544 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-b__llama3_8b__wikics__s3407:166s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:426s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:1388s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:99%260811-17:31:15 (UTC+8)
1313tasks total=14270 running=3 done=2722 failed=0 blocked=0 pending=11545 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s3407:1034s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:245s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:1207s | GPU cuda1:7851/24564MiB:94%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-17:28:12 (UTC+8)
1312tasks total=14270 running=3 done=2722 failed=0 blocked=0 pending=11545 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s3407:853s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s999:64s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:1026s | GPU cuda1:7851/24564MiB:79%;cuda2:7851/24564MiB:99%;cuda3:7349/24564MiB:100%260811-17:25:11 (UTC+8)
1311tasks total=14270 running=3 done=2721 failed=0 blocked=0 pending=11546 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s3407:672s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:2351s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:845s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-17:22:10 (UTC+8)
1310tasks total=14270 running=3 done=2721 failed=0 blocked=0 pending=11546 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s3407:491s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:2171s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:664s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:100%260811-17:19:09 (UTC+8)
1309tasks total=14270 running=3 done=2721 failed=0 blocked=0 pending=11546 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s3407:310s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:1989s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:483s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-17:16:08 (UTC+8)
1308tasks total=14270 running=3 done=2721 failed=0 blocked=0 pending=11546 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s3407:128s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:1807s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:301s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:95%260811-17:13:07 (UTC+8)
1307tasks total=14270 running=3 done=2718 failed=0 blocked=0 pending=11549 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-b__llama3_8b__wikics__s3407:823s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:1625s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s3407:119s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-17:10:05 (UTC+8)
1306tasks total=14270 running=3 done=2717 failed=0 blocked=0 pending=11550 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-b__llama3_8b__wikics__s3407:642s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:1444s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:2062s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:96%260811-17:07:03 (UTC+8)
1305tasks total=14270 running=3 done=2717 failed=0 blocked=0 pending=11550 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-b__llama3_8b__wikics__s3407:461s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:1264s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:1881s | GPU cuda1:7851/24564MiB:46%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:100%260811-17:04:02 (UTC+8)
1304tasks total=14270 running=3 done=2717 failed=0 blocked=0 pending=11550 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-b__llama3_8b__wikics__s3407:280s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:1082s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:1700s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:81%260811-17:01:01 (UTC+8)
1303tasks total=14270 running=3 done=2717 failed=0 blocked=0 pending=11550 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-b__llama3_8b__wikics__s3407:99s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:901s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:1519s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-16:58:00 (UTC+8)
1302tasks total=14270 running=3 done=2716 failed=0 blocked=0 pending=11551 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s3407:602s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:530s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:1148s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:100%260811-16:51:49 (UTC+8)
1301tasks total=14270 running=3 done=2716 failed=0 blocked=0 pending=11551 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s3407:421s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:349s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:967s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:100%260811-16:48:48 (UTC+8)
1300tasks total=14270 running=3 done=2716 failed=0 blocked=0 pending=11551 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s3407:238s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s999:167s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:784s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-16:45:47 (UTC+8)
1299tasks total=14270 running=3 done=2714 failed=0 blocked=0 pending=11553 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s3407:58s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s999:1536s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:604s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:7349/24564MiB:100%260811-16:42:45 (UTC+8)
1298tasks total=14270 running=3 done=2711 failed=0 blocked=0 pending=11556 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:2201s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s999:1355s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:423s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7349/24564MiB:98%260811-16:39:44 (UTC+8)
1297tasks total=14270 running=3 done=2711 failed=0 blocked=0 pending=11556 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:2020s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s999:1174s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:241s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-16:36:43 (UTC+8)
1296tasks total=14270 running=3 done=2711 failed=0 blocked=0 pending=11556 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:1838s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s999:992s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s3407:59s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:98%260811-16:33:42 (UTC+8)
1295tasks total=14270 running=3 done=2708 failed=0 blocked=0 pending=11559 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:1657s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s999:811s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__wikics__s3407:1141s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6809/24564MiB:98%260811-16:30:40 (UTC+8)
1294tasks total=14270 running=3 done=2708 failed=0 blocked=0 pending=11559 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:1476s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s999:630s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__wikics__s3407:960s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:6809/24564MiB:100%260811-16:27:39 (UTC+8)
1293tasks total=14270 running=3 done=2708 failed=0 blocked=0 pending=11559 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:1295s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s999:449s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__wikics__s3407:780s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6809/24564MiB:100%260811-16:24:38 (UTC+8)
1292tasks total=14270 running=3 done=2708 failed=0 blocked=0 pending=11559 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:1113s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s999:267s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__wikics__s3407:598s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6809/24564MiB:97%260811-16:21:37 (UTC+8)
1291tasks total=14270 running=3 done=2708 failed=0 blocked=0 pending=11559 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:932s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s999:85s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__wikics__s3407:416s | GPU cuda1:7851/24564MiB:99%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:100%260811-16:18:35 (UTC+8)
1290tasks total=14270 running=3 done=2707 failed=0 blocked=0 pending=11560 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:750s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s999:1449s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__wikics__s3407:235s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:100%260811-16:15:33 (UTC+8)
1289tasks total=14270 running=3 done=2707 failed=0 blocked=0 pending=11560 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:569s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s999:1268s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__wikics__s3407:54s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:97%260811-16:12:32 (UTC+8)
1288tasks total=14270 running=3 done=2706 failed=0 blocked=0 pending=11561 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:388s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s999:1087s;cuda3:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s3407:1126s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6809/24564MiB:100%260811-16:09:31 (UTC+8)
1287tasks total=14270 running=3 done=2706 failed=0 blocked=0 pending=11561 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:205s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s999:904s;cuda3:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s3407:943s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:6809/24564MiB:97%260811-16:06:30 (UTC+8)
1286tasks total=14270 running=3 done=2706 failed=0 blocked=0 pending=11561 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s3407:24s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s999:723s;cuda3:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s3407:762s | GPU cuda1:7835/24564MiB:0%;cuda2:7851/24564MiB:59%;cuda3:6809/24564MiB:97%260811-16:03:27 (UTC+8)
1285tasks total=14270 running=3 done=2705 failed=0 blocked=0 pending=11562 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:2164s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s999:542s;cuda3:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s3407:581s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:6809/24564MiB:97%260811-16:00:26 (UTC+8)
1284tasks total=14270 running=3 done=2705 failed=0 blocked=0 pending=11562 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:1982s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s999:360s;cuda3:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s3407:399s | GPU cuda1:7851/24564MiB:70%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:98%260811-15:57:25 (UTC+8)
1283tasks total=14270 running=3 done=2705 failed=0 blocked=0 pending=11562 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:1800s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s999:178s;cuda3:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s3407:217s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:98%260811-15:54:23 (UTC+8)
1282tasks total=14270 running=3 done=2703 failed=0 blocked=0 pending=11564 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:1618s;cuda2:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s999:1114s;cuda3:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s3407:36s | GPU cuda1:7851/24564MiB:96%;cuda2:4059/24564MiB:0%;cuda3:6789/24564MiB:98%260811-15:51:21 (UTC+8)
1281tasks total=14270 running=3 done=2700 failed=0 blocked=0 pending=11567 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:1437s;cuda2:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s999:933s;cuda3:trace_context:trace_bank__base__HopQuota-256-b__llama31_4b__wikics__s3407:676s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6509/24564MiB:100%260811-15:48:19 (UTC+8)
1280tasks total=14270 running=3 done=2700 failed=0 blocked=0 pending=11567 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:1257s;cuda2:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s999:752s;cuda3:trace_context:trace_bank__base__HopQuota-256-b__llama31_4b__wikics__s3407:495s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6509/24564MiB:100%260811-15:45:19 (UTC+8)
1279tasks total=14270 running=3 done=2700 failed=0 blocked=0 pending=11567 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:1076s;cuda2:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s999:571s;cuda3:trace_context:trace_bank__base__HopQuota-256-b__llama31_4b__wikics__s3407:314s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6509/24564MiB:98%260811-15:42:18 (UTC+8)
1278tasks total=14270 running=3 done=2700 failed=0 blocked=0 pending=11567 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:895s;cuda2:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s999:391s;cuda3:trace_context:trace_bank__base__HopQuota-256-b__llama31_4b__wikics__s3407:133s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:6509/24564MiB:70%260811-15:39:17 (UTC+8)
1277tasks total=14270 running=3 done=2699 failed=0 blocked=0 pending=11568 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:714s;cuda2:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s999:209s;cuda3:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__wikics__s3407:771s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6509/24564MiB:100%260811-15:36:16 (UTC+8)
1276tasks total=14270 running=3 done=2699 failed=0 blocked=0 pending=11568 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:532s;cuda2:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s999:28s;cuda3:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__wikics__s3407:589s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:6509/24564MiB:99%260811-15:33:15 (UTC+8)
1275tasks total=14270 running=3 done=2698 failed=0 blocked=0 pending=11569 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:351s;cuda2:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s999:961s;cuda3:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__wikics__s3407:408s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6509/24564MiB:98%260811-15:30:14 (UTC+8)
1274tasks total=14270 running=3 done=2698 failed=0 blocked=0 pending=11569 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s3407:171s;cuda2:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s999:780s;cuda3:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__wikics__s3407:227s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:6509/24564MiB:98%260811-15:27:13 (UTC+8)
1273tasks total=14270 running=3 done=2695 failed=0 blocked=0 pending=11572 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s3407:1040s;cuda2:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s999:599s;cuda3:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__wikics__s3407:47s | GPU cuda1:7851/24564MiB:93%;cuda2:7851/24564MiB:97%;cuda3:6509/24564MiB:99%260811-15:24:12 (UTC+8)
1272tasks total=14270 running=3 done=2692 failed=0 blocked=0 pending=11575 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s3407:859s;cuda2:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s999:418s;cuda3:trace_context:trace_bank__base__HopQuota-128-b__llama31_4b__wikics__s3407:522s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:98%260811-15:21:11 (UTC+8)
1271tasks total=14270 running=3 done=2692 failed=0 blocked=0 pending=11575 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s3407:676s;cuda2:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s999:235s;cuda3:trace_context:trace_bank__base__HopQuota-128-b__llama31_4b__wikics__s3407:339s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:98%260811-15:18:10 (UTC+8)
1270tasks total=14270 running=3 done=2692 failed=0 blocked=0 pending=11575 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s3407:495s;cuda2:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s999:54s;cuda3:trace_context:trace_bank__base__HopQuota-128-b__llama31_4b__wikics__s3407:158s | GPU cuda1:7851/24564MiB:44%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:45%260811-15:15:06 (UTC+8)
1269tasks total=14270 running=3 done=2689 failed=0 blocked=0 pending=11578 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s3407:314s;cuda2:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s999:787s;cuda3:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__wikics__s3407:631s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:6469/24564MiB:99%260811-15:12:06 (UTC+8)
1268tasks total=14270 running=3 done=2689 failed=0 blocked=0 pending=11578 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s3407:133s;cuda2:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s999:606s;cuda3:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__wikics__s3407:450s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:72%;cuda3:6469/24564MiB:98%260811-15:09:05 (UTC+8)
1267tasks total=14270 running=3 done=2688 failed=0 blocked=0 pending=11579 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s3407:999s;cuda2:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s999:425s;cuda3:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__wikics__s3407:268s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:98%;cuda3:6469/24564MiB:98%260811-15:06:04 (UTC+8)
1266tasks total=14270 running=3 done=2688 failed=0 blocked=0 pending=11579 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s3407:818s;cuda2:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s999:244s;cuda3:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__wikics__s3407:87s | GPU cuda1:7851/24564MiB:18%;cuda2:7851/24564MiB:94%;cuda3:6469/24564MiB:98%260811-15:03:02 (UTC+8)
1265tasks total=14270 running=3 done=2685 failed=0 blocked=0 pending=11582 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s3407:636s;cuda2:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s999:63s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:4334s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:7349/24564MiB:100%260811-15:00:01 (UTC+8)
1264tasks total=14270 running=3 done=2684 failed=0 blocked=0 pending=11583 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s3407:456s;cuda2:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s999:790s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:4153s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:100%260811-14:57:00 (UTC+8)
1263tasks total=14270 running=3 done=2684 failed=0 blocked=0 pending=11583 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s3407:274s;cuda2:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s999:609s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:3972s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:46%260811-14:53:59 (UTC+8)
1262tasks total=14270 running=3 done=2684 failed=0 blocked=0 pending=11583 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s3407:90s;cuda2:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s999:425s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:3788s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:7349/24564MiB:46%260811-14:50:58 (UTC+8)
1261tasks total=14270 running=3 done=2681 failed=0 blocked=0 pending=11586 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-b__llama3_8b__wikics__s3407:786s;cuda2:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s999:244s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:3607s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:94%;cuda3:7349/24564MiB:100%260811-14:47:54 (UTC+8)
1260tasks total=14270 running=3 done=2681 failed=0 blocked=0 pending=11586 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-b__llama3_8b__wikics__s3407:603s;cuda2:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s999:62s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:3425s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:100%260811-14:44:53 (UTC+8)
1259tasks total=14270 running=3 done=2679 failed=0 blocked=0 pending=11588 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-b__llama3_8b__wikics__s3407:423s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:2311s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:3244s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:46%260811-14:41:51 (UTC+8)
1258tasks total=14270 running=3 done=2679 failed=0 blocked=0 pending=11588 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-b__llama3_8b__wikics__s3407:242s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:2130s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:3063s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:96%260811-14:38:50 (UTC+8)
1257tasks total=14270 running=3 done=2679 failed=0 blocked=0 pending=11588 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-b__llama3_8b__wikics__s3407:61s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:1949s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:2882s | GPU cuda1:7851/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-14:35:49 (UTC+8)
1256tasks total=14270 running=3 done=2678 failed=0 blocked=0 pending=11589 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s3407:752s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:1768s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:2702s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:48%260811-14:32:48 (UTC+8)
1255tasks total=14270 running=3 done=2678 failed=0 blocked=0 pending=11589 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s3407:571s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:1587s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:2521s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:49%260811-14:29:47 (UTC+8)
1254tasks total=14270 running=3 done=2678 failed=0 blocked=0 pending=11589 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s3407:385s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:1402s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:2335s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:46%260811-14:26:46 (UTC+8)
1253tasks total=14270 running=3 done=2678 failed=0 blocked=0 pending=11589 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s3407:204s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:1220s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:2154s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:96%260811-14:23:41 (UTC+8)
1252tasks total=14270 running=3 done=2678 failed=0 blocked=0 pending=11589 | results done=889/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s3407:23s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:1039s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:1973s | GPU cuda1:7835/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:96%260811-14:20:40 (UTC+8)
1251tasks total=14270 running=3 done=2669 failed=0 blocked=0 pending=11598 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s3407:1047s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:858s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:1791s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:48%260811-14:17:38 (UTC+8)
1250tasks total=14270 running=3 done=2669 failed=0 blocked=0 pending=11598 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s3407:866s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:677s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:1610s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:46%260811-14:14:37 (UTC+8)
1249tasks total=14270 running=3 done=2669 failed=0 blocked=0 pending=11598 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s3407:685s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:496s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:1430s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:74%260811-14:11:36 (UTC+8)
1248tasks total=14270 running=3 done=2669 failed=0 blocked=0 pending=11598 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s3407:504s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:315s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:1249s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:100%260811-14:08:35 (UTC+8)
1247tasks total=14270 running=3 done=2669 failed=0 blocked=0 pending=11598 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s3407:323s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s999:134s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:1068s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-14:05:35 (UTC+8)
1246tasks total=14270 running=3 done=2668 failed=0 blocked=0 pending=11599 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s3407:142s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:2376s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:887s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:46%260811-14:02:34 (UTC+8)
1245tasks total=14270 running=3 done=2667 failed=0 blocked=0 pending=11600 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s3407:1038s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:2195s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:706s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:48%260811-13:59:32 (UTC+8)
1244tasks total=14270 running=3 done=2667 failed=0 blocked=0 pending=11600 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s3407:856s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:2013s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:524s | GPU cuda1:7851/24564MiB:94%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:96%260811-13:56:32 (UTC+8)
1243tasks total=14270 running=3 done=2667 failed=0 blocked=0 pending=11600 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s3407:675s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:1832s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:342s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-13:53:30 (UTC+8)
1242tasks total=14270 running=3 done=2667 failed=0 blocked=0 pending=11600 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s3407:494s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:1651s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s3407:162s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:51%260811-13:50:28 (UTC+8)
1241tasks total=14270 running=3 done=2666 failed=0 blocked=0 pending=11601 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s3407:313s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:1470s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:3784s | GPU cuda1:7851/24564MiB:64%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:55%260811-13:47:27 (UTC+8)
1240tasks total=14270 running=3 done=2666 failed=0 blocked=0 pending=11601 | results done=883/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s3407:132s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:1289s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:3603s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:69%260811-13:44:26 (UTC+8)
1239tasks total=14270 running=3 done=2661 failed=0 blocked=0 pending=11606 | results done=879/5050 | RUNNING cuda1:fixed_downstream:run__T1_FixedFull512__llama3_8b__wikics__s3407__gcn3:31s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:1108s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:3422s | GPU cuda1:1637/24564MiB:80%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:96%260811-13:41:26 (UTC+8)
1238tasks total=14270 running=3 done=2657 failed=0 blocked=0 pending=11610 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s3407:1344s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:927s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:3241s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-13:38:25 (UTC+8)
1237tasks total=14270 running=3 done=2657 failed=0 blocked=0 pending=11610 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s3407:1163s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:746s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:3059s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:64%260811-13:35:24 (UTC+8)
1236tasks total=14270 running=3 done=2657 failed=0 blocked=0 pending=11610 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s3407:982s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:565s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:2878s | GPU cuda1:7851/24564MiB:99%;cuda2:7851/24564MiB:96%;cuda3:5445/24564MiB:94%260811-13:32:22 (UTC+8)
1235tasks total=14270 running=3 done=2657 failed=0 blocked=0 pending=11610 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s3407:801s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:384s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:2698s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:5445/24564MiB:0%260811-13:29:21 (UTC+8)
1234tasks total=14270 running=3 done=2657 failed=0 blocked=0 pending=11610 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s3407:619s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:202s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:2516s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:5445/24564MiB:95%260811-13:26:20 (UTC+8)
1233tasks total=14270 running=3 done=2657 failed=0 blocked=0 pending=11610 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s3407:438s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s999:21s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:2334s | GPU cuda1:7851/24564MiB:96%;cuda2:7727/24564MiB:14%;cuda3:5445/24564MiB:0%260811-13:23:19 (UTC+8)
1232tasks total=14270 running=3 done=2655 failed=0 blocked=0 pending=11612 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s3407:257s;cuda2:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s999:942s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:2154s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:100%260811-13:20:17 (UTC+8)
1231tasks total=14270 running=3 done=2655 failed=0 blocked=0 pending=11612 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s3407:75s;cuda2:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s999:760s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:1972s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:0%260811-13:17:16 (UTC+8)
1230tasks total=14270 running=3 done=2654 failed=0 blocked=0 pending=11613 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s3407:1349s;cuda2:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s999:578s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:1789s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:69%;cuda3:5445/24564MiB:100%260811-13:14:14 (UTC+8)
1229tasks total=14270 running=3 done=2654 failed=0 blocked=0 pending=11613 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s3407:1168s;cuda2:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s999:397s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:1609s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:5445/24564MiB:0%260811-13:11:12 (UTC+8)
1228tasks total=14270 running=3 done=2654 failed=0 blocked=0 pending=11613 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s3407:987s;cuda2:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s999:216s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:1428s | GPU cuda1:7851/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:0%260811-13:08:11 (UTC+8)
1227tasks total=14270 running=3 done=2654 failed=0 blocked=0 pending=11613 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s3407:806s;cuda2:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s999:35s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:1247s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:94%260811-13:05:11 (UTC+8)
1226tasks total=14270 running=3 done=2653 failed=0 blocked=0 pending=11614 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s3407:625s;cuda2:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s999:953s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:1065s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:5445/24564MiB:0%260811-13:02:10 (UTC+8)
1225tasks total=14270 running=3 done=2653 failed=0 blocked=0 pending=11614 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s3407:444s;cuda2:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s999:772s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:884s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:5445/24564MiB:100%260811-12:59:08 (UTC+8)
1224tasks total=14270 running=3 done=2653 failed=0 blocked=0 pending=11614 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s3407:263s;cuda2:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s999:591s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:703s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:0%260811-12:56:07 (UTC+8)
1223tasks total=14270 running=3 done=2653 failed=0 blocked=0 pending=11614 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s3407:82s;cuda2:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s999:410s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:523s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:95%260811-12:53:06 (UTC+8)
1222tasks total=14270 running=3 done=2652 failed=0 blocked=0 pending=11615 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Self__llama3_8b__wikics__s3407:623s;cuda2:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s999:229s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:341s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:98%;cuda3:5445/24564MiB:0%260811-12:50:05 (UTC+8)
1221tasks total=14270 running=3 done=2652 failed=0 blocked=0 pending=11615 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Self__llama3_8b__wikics__s3407:441s;cuda2:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s999:47s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s3407:159s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:5445/24564MiB:0%260811-12:47:04 (UTC+8)
1220tasks total=14270 running=3 done=2647 failed=0 blocked=0 pending=11620 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Self__llama3_8b__wikics__s3407:260s;cuda2:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s999:772s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:3108s | GPU cuda1:7851/24564MiB:98%;cuda2:7851/24564MiB:95%;cuda3:4885/24564MiB:100%260811-12:44:02 (UTC+8)
1219tasks total=14270 running=3 done=2647 failed=0 blocked=0 pending=11620 | results done=877/5050 | RUNNING cuda1:trace_context:trace_bank__base__Self__llama3_8b__wikics__s3407:79s;cuda2:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s999:591s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:2927s | GPU cuda1:7851/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:0%260811-12:41:01 (UTC+8)
1218tasks total=14270 running=3 done=2646 failed=0 blocked=0 pending=11621 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:3837s;cuda2:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s999:410s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:2746s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:94%;cuda3:4885/24564MiB:0%260811-12:38:00 (UTC+8)
1217tasks total=14270 running=3 done=2646 failed=0 blocked=0 pending=11621 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:3656s;cuda2:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s999:229s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:2565s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:0%260811-12:34:59 (UTC+8)
1216tasks total=14270 running=3 done=2646 failed=0 blocked=0 pending=11621 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:3475s;cuda2:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s999:48s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:2384s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:4885/24564MiB:0%260811-12:31:58 (UTC+8)
1215tasks total=14270 running=3 done=2645 failed=0 blocked=0 pending=11622 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:3293s;cuda2:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s999:772s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:2202s | GPU cuda1:10479/24564MiB:99%;cuda2:7851/24564MiB:98%;cuda3:4885/24564MiB:96%260811-12:28:56 (UTC+8)
1214tasks total=14270 running=3 done=2645 failed=0 blocked=0 pending=11622 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:3113s;cuda2:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s999:591s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:2021s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:4885/24564MiB:0%260811-12:25:55 (UTC+8)
1213tasks total=14270 running=3 done=2645 failed=0 blocked=0 pending=11622 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:2932s;cuda2:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s999:410s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:1841s | GPU cuda1:10479/24564MiB:94%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:96%260811-12:22:54 (UTC+8)
1212tasks total=14270 running=3 done=2645 failed=0 blocked=0 pending=11622 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:2751s;cuda2:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s999:229s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:1659s | GPU cuda1:10479/24564MiB:99%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:0%260811-12:19:53 (UTC+8)
1211tasks total=14270 running=3 done=2645 failed=0 blocked=0 pending=11622 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:2570s;cuda2:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s999:48s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:1479s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:4885/24564MiB:96%260811-12:16:52 (UTC+8)
1210tasks total=14270 running=3 done=2643 failed=0 blocked=0 pending=11624 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:2389s;cuda2:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s999:995s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:1298s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:4885/24564MiB:0%260811-12:13:51 (UTC+8)
1209tasks total=14270 running=3 done=2643 failed=0 blocked=0 pending=11624 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:2207s;cuda2:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s999:813s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:1116s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:27%260811-12:10:51 (UTC+8)
1208tasks total=14270 running=3 done=2643 failed=0 blocked=0 pending=11624 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:2027s;cuda2:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s999:633s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:936s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:4885/24564MiB:0%260811-12:07:49 (UTC+8)
1207tasks total=14270 running=3 done=2643 failed=0 blocked=0 pending=11624 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:1846s;cuda2:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s999:452s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:754s | GPU cuda1:10479/24564MiB:99%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:54%260811-12:04:48 (UTC+8)
1206tasks total=14270 running=3 done=2643 failed=0 blocked=0 pending=11624 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:1665s;cuda2:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s999:271s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:574s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:97%260811-12:01:47 (UTC+8)
1205tasks total=14270 running=3 done=2643 failed=0 blocked=0 pending=11624 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:1484s;cuda2:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s999:90s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:393s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:46%260811-11:58:47 (UTC+8)
1204tasks total=14270 running=3 done=2642 failed=0 blocked=0 pending=11625 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:1303s;cuda2:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s999:1035s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:212s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:32%260811-11:55:45 (UTC+8)
1203tasks total=14270 running=3 done=2642 failed=0 blocked=0 pending=11625 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:1122s;cuda2:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s999:854s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s3407:31s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6453/24564MiB:46%260811-11:52:45 (UTC+8)
1202tasks total=14270 running=3 done=2641 failed=0 blocked=0 pending=11626 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:941s;cuda2:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s999:673s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:3341s | GPU cuda1:10479/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:46%260811-11:49:44 (UTC+8)
1201tasks total=14270 running=3 done=2641 failed=0 blocked=0 pending=11626 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:760s;cuda2:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s999:492s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:3160s | GPU cuda1:10477/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:100%260811-11:46:42 (UTC+8)
1200tasks total=14270 running=3 done=2641 failed=0 blocked=0 pending=11626 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:579s;cuda2:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s999:311s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:2979s | GPU cuda1:10477/24564MiB:99%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:46%260811-11:43:42 (UTC+8)
1199tasks total=14270 running=3 done=2641 failed=0 blocked=0 pending=11626 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:398s;cuda2:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s999:131s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:2798s | GPU cuda1:10477/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:47%260811-11:40:41 (UTC+8)
1198tasks total=14270 running=3 done=2639 failed=0 blocked=0 pending=11628 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:217s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s999:1469s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:2617s | GPU cuda1:10477/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:97%260811-11:37:40 (UTC+8)
1197tasks total=14270 running=3 done=2639 failed=0 blocked=0 pending=11628 | results done=877/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s3407:36s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s999:1287s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:2436s | GPU cuda1:401/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:46%260811-11:34:39 (UTC+8)
1196tasks total=14270 running=3 done=2633 failed=0 blocked=0 pending=11634 | results done=872/5050 | RUNNING cuda1:external_downstream:run__LinguGKD__llama31_4b__wikics__s3407__gcn2:18s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s999:1107s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:2255s | GPU cuda1:1983/24564MiB:29%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:50%260811-11:31:38 (UTC+8)
1195tasks total=14270 running=3 done=2630 failed=0 blocked=0 pending=11637 | results done=871/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s3407:1463s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s999:926s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:2074s | GPU cuda1:5545/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:48%260811-11:28:37 (UTC+8)
1194tasks total=14270 running=3 done=2630 failed=0 blocked=0 pending=11637 | results done=871/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s3407:1282s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s999:745s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:1893s | GPU cuda1:5545/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:97%260811-11:25:36 (UTC+8)
1193tasks total=14270 running=3 done=2630 failed=0 blocked=0 pending=11637 | results done=871/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s3407:1102s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s999:564s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:1712s | GPU cuda1:5545/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:54%260811-11:22:35 (UTC+8)
1192tasks total=14270 running=3 done=2630 failed=0 blocked=0 pending=11637 | results done=871/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s3407:920s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s999:383s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:1531s | GPU cuda1:5545/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:48%260811-11:19:34 (UTC+8)
1191tasks total=14270 running=3 done=2630 failed=0 blocked=0 pending=11637 | results done=871/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s3407:739s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s999:202s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:1350s | GPU cuda1:5545/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:49%260811-11:16:33 (UTC+8)
1190tasks total=14270 running=3 done=2630 failed=0 blocked=0 pending=11637 | results done=871/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s3407:559s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s999:21s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:1170s | GPU cuda1:5545/24564MiB:100%;cuda2:7731/24564MiB:13%;cuda3:6789/24564MiB:55%260811-11:13:32 (UTC+8)
1189tasks total=14270 running=3 done=2629 failed=0 blocked=0 pending=11638 | results done=871/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s3407:378s;cuda2:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s999:1361s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:989s | GPU cuda1:5545/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:100%260811-11:10:31 (UTC+8)
1188tasks total=14270 running=3 done=2629 failed=0 blocked=0 pending=11638 | results done=871/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s3407:197s;cuda2:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s999:1180s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:808s | GPU cuda1:5545/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:97%260811-11:07:31 (UTC+8)
1187tasks total=14270 running=3 done=2629 failed=0 blocked=0 pending=11638 | results done=871/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s3407:16s;cuda2:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s999:999s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:626s | GPU cuda1:4549/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:49%260811-11:04:29 (UTC+8)
1186tasks total=14270 running=3 done=2628 failed=0 blocked=0 pending=11639 | results done=871/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:4626s;cuda2:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s999:817s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:444s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:46%260811-11:01:28 (UTC+8)
1185tasks total=14270 running=3 done=2628 failed=0 blocked=0 pending=11639 | results done=871/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:4445s;cuda2:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s999:636s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:264s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:49%260811-10:58:26 (UTC+8)
1184tasks total=14270 running=3 done=2628 failed=0 blocked=0 pending=11639 | results done=871/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:4264s;cuda2:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s999:455s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s3407:83s | GPU cuda1:9171/24564MiB:99%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:97%260811-10:55:25 (UTC+8)
1183tasks total=14270 running=3 done=2625 failed=0 blocked=0 pending=11642 | results done=871/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:4084s;cuda2:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s999:274s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:3014s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:100%260811-10:52:25 (UTC+8)
1182tasks total=14270 running=3 done=2625 failed=0 blocked=0 pending=11642 | results done=871/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:3902s;cuda2:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s999:93s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:2832s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:45%260811-10:49:24 (UTC+8)
1181tasks total=14270 running=3 done=2623 failed=0 blocked=0 pending=11644 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:3720s;cuda2:trace_analysis:trace_analysis__n1__llama3_8b__wikics__s999:0s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:2650s | GPU cuda1:9171/24564MiB:100%;cuda2:14/24564MiB:0%;cuda3:6529/24564MiB:65%260811-10:46:22 (UTC+8)
1180tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:3538s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:2086s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:2468s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:99%260811-10:43:20 (UTC+8)
1179tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:3357s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:1906s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:2287s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:48%260811-10:40:18 (UTC+8)
1178tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:3176s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:1725s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:2106s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:49%260811-10:37:17 (UTC+8)
1177tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:2995s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:1544s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:1925s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:98%260811-10:34:16 (UTC+8)
1176tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:2814s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:1363s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:1744s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:48%260811-10:31:15 (UTC+8)
1175tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:2633s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:1182s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:1563s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:98%260811-10:28:14 (UTC+8)
1174tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:2451s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:1000s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:1381s | GPU cuda1:9171/24564MiB:99%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:46%260811-10:25:13 (UTC+8)
1173tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:2270s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:819s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:1200s | GPU cuda1:9171/24564MiB:99%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:51%260811-10:22:11 (UTC+8)
1172tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:2089s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:638s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:1019s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:98%260811-10:19:11 (UTC+8)
1171tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:1909s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:457s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:839s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:46%260811-10:16:10 (UTC+8)
1170tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:1728s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:277s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:658s | GPU cuda1:9171/24564MiB:99%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:98%260811-10:13:09 (UTC+8)
1169tasks total=14270 running=3 done=2621 failed=0 blocked=0 pending=11646 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:1547s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s999:96s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:477s | GPU cuda1:9171/24564MiB:99%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:68%260811-10:10:08 (UTC+8)
1168tasks total=14270 running=3 done=2620 failed=0 blocked=0 pending=11647 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:1365s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:2170s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:296s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:47%260811-10:07:07 (UTC+8)
1167tasks total=14270 running=3 done=2620 failed=0 blocked=0 pending=11647 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:1185s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:1989s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s3407:115s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:100%260811-10:04:06 (UTC+8)
1166tasks total=14270 running=3 done=2619 failed=0 blocked=0 pending=11648 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:1004s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:1809s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:2974s | GPU cuda1:9171/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:6529/24564MiB:46%260811-10:01:05 (UTC+8)
1165tasks total=14270 running=3 done=2619 failed=0 blocked=0 pending=11648 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:821s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:1626s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:2791s | GPU cuda1:9169/24564MiB:99%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:98%260811-09:58:04 (UTC+8)
1164tasks total=14270 running=3 done=2619 failed=0 blocked=0 pending=11648 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:640s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:1445s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:2610s | GPU cuda1:8489/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:46%260811-09:55:01 (UTC+8)
1163tasks total=14270 running=3 done=2619 failed=0 blocked=0 pending=11648 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:459s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:1264s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:2429s | GPU cuda1:8489/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:46%260811-09:52:00 (UTC+8)
1162tasks total=14270 running=3 done=2619 failed=0 blocked=0 pending=11648 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:278s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:1083s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:2248s | GPU cuda1:8489/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:100%260811-09:48:59 (UTC+8)
1161tasks total=14270 running=3 done=2619 failed=0 blocked=0 pending=11648 | results done=870/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s3407:97s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:902s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:2067s | GPU cuda1:401/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:53%260811-09:45:59 (UTC+8)
1160tasks total=14270 running=3 done=2616 failed=0 blocked=0 pending=11651 | results done=869/5050 | RUNNING cuda1:trace_analysis:trace_analysis__n2__llama31_4b__wikics__s999:18s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:721s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:1886s | GPU cuda1:1631/24564MiB:87%;cuda2:7851/24564MiB:96%;cuda3:6529/24564MiB:98%260811-09:42:57 (UTC+8)
1159tasks total=14270 running=3 done=2614 failed=0 blocked=0 pending=11653 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s999:1301s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:539s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:1704s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:45%260811-09:39:56 (UTC+8)
1158tasks total=14270 running=3 done=2614 failed=0 blocked=0 pending=11653 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s999:1120s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:358s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:1523s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6529/24564MiB:98%260811-09:36:55 (UTC+8)
1157tasks total=14270 running=3 done=2614 failed=0 blocked=0 pending=11653 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s999:939s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s999:177s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:1342s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:6529/24564MiB:46%260811-09:33:53 (UTC+8)
1156tasks total=14270 running=3 done=2612 failed=0 blocked=0 pending=11655 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s999:758s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s999:1404s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:1161s | GPU cuda1:5445/24564MiB:100%;cuda2:4079/24564MiB:0%;cuda3:6529/24564MiB:14%260811-09:30:53 (UTC+8)
1155tasks total=14270 running=3 done=2612 failed=0 blocked=0 pending=11655 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s999:577s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s999:1223s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:980s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:48%260811-09:27:52 (UTC+8)
1154tasks total=14270 running=3 done=2612 failed=0 blocked=0 pending=11655 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s999:396s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s999:1042s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:799s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:64%260811-09:24:51 (UTC+8)
1153tasks total=14270 running=3 done=2612 failed=0 blocked=0 pending=11655 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s999:215s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s999:861s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:618s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:47%260811-09:21:49 (UTC+8)
1152tasks total=14270 running=3 done=2612 failed=0 blocked=0 pending=11655 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s999:34s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s999:680s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:437s | GPU cuda1:5425/24564MiB:0%;cuda2:7851/24564MiB:98%;cuda3:6529/24564MiB:98%260811-09:18:49 (UTC+8)
1151tasks total=14270 running=3 done=2611 failed=0 blocked=0 pending=11656 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s999:1315s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s999:499s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:257s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:48%260811-09:15:48 (UTC+8)
1150tasks total=14270 running=3 done=2611 failed=0 blocked=0 pending=11656 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s999:1134s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s999:318s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s3407:76s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:98%260811-09:12:47 (UTC+8)
1149tasks total=14270 running=3 done=2608 failed=0 blocked=0 pending=11659 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s999:953s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s999:137s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:2834s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:46%260811-09:09:46 (UTC+8)
1148tasks total=14270 running=3 done=2607 failed=0 blocked=0 pending=11660 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s999:772s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s999:1364s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:2653s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:46%260811-09:06:45 (UTC+8)
1147tasks total=14270 running=3 done=2607 failed=0 blocked=0 pending=11660 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s999:591s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s999:1183s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:2472s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:45%260811-09:03:44 (UTC+8)
1146tasks total=14270 running=3 done=2607 failed=0 blocked=0 pending=11660 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s999:410s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s999:1002s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:2291s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:44%260811-09:00:43 (UTC+8)
1145tasks total=14270 running=3 done=2607 failed=0 blocked=0 pending=11660 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s999:227s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s999:820s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:2109s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:98%260811-08:57:42 (UTC+8)
1144tasks total=14270 running=3 done=2607 failed=0 blocked=0 pending=11660 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s999:46s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s999:638s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:1927s | GPU cuda1:5425/24564MiB:94%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:44%260811-08:54:39 (UTC+8)
1143tasks total=14270 running=3 done=2605 failed=0 blocked=0 pending=11662 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s999:748s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s999:458s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:1747s | GPU cuda1:4905/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6469/24564MiB:45%260811-08:51:38 (UTC+8)
1142tasks total=14270 running=3 done=2605 failed=0 blocked=0 pending=11662 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s999:568s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s999:277s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:1566s | GPU cuda1:4905/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:98%260811-08:48:37 (UTC+8)
1141tasks total=14270 running=3 done=2605 failed=0 blocked=0 pending=11662 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s999:387s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s999:96s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:1385s | GPU cuda1:4905/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:44%260811-08:45:36 (UTC+8)
1140tasks total=14270 running=3 done=2603 failed=0 blocked=0 pending=11664 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s999:206s;cuda2:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s999:881s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:1204s | GPU cuda1:4905/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:42%260811-08:42:36 (UTC+8)
1139tasks total=14270 running=3 done=2603 failed=0 blocked=0 pending=11664 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s999:25s;cuda2:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s999:700s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:1023s | GPU cuda1:4885/24564MiB:98%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:45%260811-08:39:34 (UTC+8)
1138tasks total=14270 running=3 done=2602 failed=0 blocked=0 pending=11665 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s999:726s;cuda2:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s999:519s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:842s | GPU cuda1:4905/24564MiB:96%;cuda2:7851/24564MiB:98%;cuda3:6469/24564MiB:45%260811-08:36:34 (UTC+8)
1137tasks total=14270 running=3 done=2602 failed=0 blocked=0 pending=11665 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s999:546s;cuda2:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s999:338s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:661s | GPU cuda1:4905/24564MiB:96%;cuda2:7851/24564MiB:98%;cuda3:6469/24564MiB:66%260811-08:33:33 (UTC+8)
1136tasks total=14270 running=3 done=2602 failed=0 blocked=0 pending=11665 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s999:365s;cuda2:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s999:157s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:480s | GPU cuda1:4905/24564MiB:96%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:44%260811-08:30:32 (UTC+8)
1135tasks total=14270 running=3 done=2601 failed=0 blocked=0 pending=11666 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s999:183s;cuda2:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s999:941s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:299s | GPU cuda1:4905/24564MiB:96%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:99%260811-08:27:31 (UTC+8)
1134tasks total=14270 running=3 done=2601 failed=0 blocked=0 pending=11666 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s999:2s;cuda2:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s999:760s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s3407:118s | GPU cuda1:401/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:46%260811-08:24:30 (UTC+8)
1133tasks total=14270 running=3 done=2598 failed=0 blocked=0 pending=11669 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-256-b__llama31_4b__wikics__s999:423s;cuda2:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s999:579s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:2870s | GPU cuda1:4605/24564MiB:94%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:41%260811-08:21:29 (UTC+8)
1132tasks total=14270 running=3 done=2598 failed=0 blocked=0 pending=11669 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-256-b__llama31_4b__wikics__s999:243s;cuda2:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s999:398s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:2689s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:7%;cuda3:6469/24564MiB:98%260811-08:18:28 (UTC+8)
1131tasks total=14270 running=3 done=2598 failed=0 blocked=0 pending=11669 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-256-b__llama31_4b__wikics__s999:62s;cuda2:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s999:217s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:2508s | GPU cuda1:4605/24564MiB:94%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:46%260811-08:15:27 (UTC+8)
1130tasks total=14270 running=3 done=2597 failed=0 blocked=0 pending=11670 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s999:480s;cuda2:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s999:36s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:2327s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:63%260811-08:12:26 (UTC+8)
1129tasks total=14270 running=3 done=2595 failed=0 blocked=0 pending=11672 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s999:299s;cuda2:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s999:681s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:2146s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:100%260811-08:09:25 (UTC+8)
1128tasks total=14270 running=3 done=2595 failed=0 blocked=0 pending=11672 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s999:119s;cuda2:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s999:500s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:1965s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:94%;cuda3:6469/24564MiB:44%260811-08:06:24 (UTC+8)
1127tasks total=14270 running=3 done=2593 failed=0 blocked=0 pending=11674 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-128-b__llama31_4b__wikics__s999:400s;cuda2:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s999:319s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:1784s | GPU cuda1:4565/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:44%260811-08:03:23 (UTC+8)
1126tasks total=14270 running=3 done=2593 failed=0 blocked=0 pending=11674 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-128-b__llama31_4b__wikics__s999:219s;cuda2:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s999:138s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:1603s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:45%260811-08:00:22 (UTC+8)
1125tasks total=14270 running=3 done=2592 failed=0 blocked=0 pending=11675 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-128-b__llama31_4b__wikics__s999:38s;cuda2:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s999:776s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:1422s | GPU cuda1:4565/24564MiB:5%;cuda2:7851/24564MiB:98%;cuda3:6469/24564MiB:45%260811-07:57:21 (UTC+8)
1124tasks total=14270 running=3 done=2591 failed=0 blocked=0 pending=11676 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-128-a__llama31_4b__wikics__s999:318s;cuda2:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s999:595s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:1241s | GPU cuda1:4565/24564MiB:7%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:45%260811-07:54:20 (UTC+8)
1123tasks total=14270 running=3 done=2591 failed=0 blocked=0 pending=11676 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-128-a__llama31_4b__wikics__s999:137s;cuda2:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s999:414s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:1060s | GPU cuda1:4565/24564MiB:94%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:99%260811-07:51:19 (UTC+8)
1122tasks total=14270 running=3 done=2589 failed=0 blocked=0 pending=11678 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:3641s;cuda2:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s999:233s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:879s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:6469/24564MiB:45%260811-07:48:18 (UTC+8)
1121tasks total=14270 running=3 done=2589 failed=0 blocked=0 pending=11678 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:3460s;cuda2:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s999:52s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:698s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:46%260811-07:45:17 (UTC+8)
1120tasks total=14270 running=3 done=2587 failed=0 blocked=0 pending=11680 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:3279s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:4670s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:517s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:45%260811-07:42:16 (UTC+8)
1119tasks total=14270 running=3 done=2587 failed=0 blocked=0 pending=11680 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:3098s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:4490s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:337s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:6469/24564MiB:100%260811-07:39:15 (UTC+8)
1118tasks total=14270 running=3 done=2587 failed=0 blocked=0 pending=11680 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:2917s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:4308s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s3407:155s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:46%260811-07:36:14 (UTC+8)
1117tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:2736s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:4127s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:2292s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-07:33:13 (UTC+8)
1116tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:2555s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:3946s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:2111s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-07:30:12 (UTC+8)
1115tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:2374s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:3765s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:1930s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-07:27:11 (UTC+8)
1114tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:2193s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:3585s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:1749s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-07:24:10 (UTC+8)
1113tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:2012s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:3403s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:1568s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-07:21:09 (UTC+8)
1112tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:1831s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:3222s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:1387s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-07:18:08 (UTC+8)
1111tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:1650s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:3041s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:1206s | GPU cuda1:5465/24564MiB:49%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:98%260811-07:15:07 (UTC+8)
1110tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:1469s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:2860s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:1024s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-07:12:06 (UTC+8)
1109tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:1288s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:2679s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:844s | GPU cuda1:5465/24564MiB:94%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:96%260811-07:09:05 (UTC+8)
1108tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:1107s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:2498s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:662s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-07:06:04 (UTC+8)
1107tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:926s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:2317s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:482s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:98%260811-07:03:03 (UTC+8)
1106tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:745s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:2136s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:300s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-07:00:02 (UTC+8)
1105tasks total=14270 running=3 done=2584 failed=0 blocked=0 pending=11683 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:564s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:1955s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s3407:120s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-06:57:01 (UTC+8)
1104tasks total=14270 running=3 done=2583 failed=0 blocked=0 pending=11684 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:383s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:1774s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:2275s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-06:54:00 (UTC+8)
1103tasks total=14270 running=3 done=2583 failed=0 blocked=0 pending=11684 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:202s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:1593s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:2094s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:92%260811-06:50:59 (UTC+8)
1102tasks total=14270 running=3 done=2583 failed=0 blocked=0 pending=11684 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s999:21s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:1412s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:1912s | GPU cuda1:4549/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-06:47:58 (UTC+8)
1101tasks total=14270 running=3 done=2582 failed=0 blocked=0 pending=11685 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:3540s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:1231s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:1732s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-06:44:57 (UTC+8)
1100tasks total=14270 running=3 done=2582 failed=0 blocked=0 pending=11685 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:3359s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:1051s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:1551s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-06:41:56 (UTC+8)
1099tasks total=14270 running=3 done=2582 failed=0 blocked=0 pending=11685 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:3178s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:870s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:1370s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:97%260811-06:38:55 (UTC+8)
1098tasks total=14270 running=3 done=2582 failed=0 blocked=0 pending=11685 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:2995s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:687s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:1187s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:96%260811-06:35:54 (UTC+8)
1097tasks total=14270 running=3 done=2582 failed=0 blocked=0 pending=11685 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:2815s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:506s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:1007s | GPU cuda1:5465/24564MiB:94%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-06:32:52 (UTC+8)
1096tasks total=14270 running=3 done=2582 failed=0 blocked=0 pending=11685 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:2634s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:326s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:826s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-06:29:51 (UTC+8)
1095tasks total=14270 running=3 done=2582 failed=0 blocked=0 pending=11685 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:2453s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s999:145s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:645s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-06:26:50 (UTC+8)
1094tasks total=14270 running=3 done=2581 failed=0 blocked=0 pending=11686 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:2272s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:4755s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:464s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:100%260811-06:23:49 (UTC+8)
1093tasks total=14270 running=3 done=2581 failed=0 blocked=0 pending=11686 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:2091s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:4574s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:283s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-06:20:48 (UTC+8)
1092tasks total=14270 running=3 done=2581 failed=0 blocked=0 pending=11686 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:1910s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:4394s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s3407:102s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:96%260811-06:17:47 (UTC+8)
1091tasks total=14270 running=3 done=2578 failed=0 blocked=0 pending=11689 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:1729s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:4213s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s3407:1279s | GPU cuda1:5465/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:100%260811-06:14:46 (UTC+8)
1090tasks total=14270 running=3 done=2578 failed=0 blocked=0 pending=11689 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:1548s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:4032s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s3407:1098s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:99%260811-06:11:46 (UTC+8)
1089tasks total=14270 running=3 done=2578 failed=0 blocked=0 pending=11689 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:1367s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:3851s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s3407:917s | GPU cuda1:5465/24564MiB:93%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:100%260811-06:08:45 (UTC+8)
1088tasks total=14270 running=3 done=2578 failed=0 blocked=0 pending=11689 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:1186s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:3670s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s3407:736s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260811-06:05:44 (UTC+8)
1087tasks total=14270 running=3 done=2578 failed=0 blocked=0 pending=11689 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:1005s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:3489s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s3407:555s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:100%260811-06:02:42 (UTC+8)
1086tasks total=14270 running=3 done=2578 failed=0 blocked=0 pending=11689 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:824s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:3308s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s3407:374s | GPU cuda1:5465/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260811-05:59:42 (UTC+8)
1085tasks total=14270 running=3 done=2578 failed=0 blocked=0 pending=11689 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:644s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:3127s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s3407:193s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:100%260811-05:56:41 (UTC+8)
1084tasks total=14270 running=3 done=2578 failed=0 blocked=0 pending=11689 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:463s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:2946s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s3407:12s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6347/24564MiB:64%260811-05:53:40 (UTC+8)
1083tasks total=14270 running=3 done=2577 failed=0 blocked=0 pending=11690 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:281s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:2765s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s3407:1184s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:100%260811-05:50:39 (UTC+8)
1082tasks total=14270 running=3 done=2577 failed=0 blocked=0 pending=11690 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s999:100s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:2584s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s3407:1003s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260811-05:47:38 (UTC+8)
1081tasks total=14270 running=3 done=2575 failed=0 blocked=0 pending=11692 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:2968s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:2403s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s3407:822s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:97%260811-05:44:37 (UTC+8)
1080tasks total=14270 running=3 done=2575 failed=0 blocked=0 pending=11692 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:2787s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:2222s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s3407:641s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260811-05:41:36 (UTC+8)
1079tasks total=14270 running=3 done=2575 failed=0 blocked=0 pending=11692 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:2605s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:2040s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s3407:459s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:100%260811-05:38:35 (UTC+8)
1078tasks total=14270 running=3 done=2575 failed=0 blocked=0 pending=11692 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:2424s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:1859s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s3407:278s | GPU cuda1:4885/24564MiB:99%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:100%260811-05:35:33 (UTC+8)
1077tasks total=14270 running=3 done=2575 failed=0 blocked=0 pending=11692 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:2243s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:1678s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s3407:97s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:99%260811-05:32:32 (UTC+8)
1076tasks total=14270 running=3 done=2572 failed=0 blocked=0 pending=11695 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:2062s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:1497s;cuda3:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__wikics__s3407:681s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:100%260811-05:29:31 (UTC+8)
1075tasks total=14270 running=3 done=2572 failed=0 blocked=0 pending=11695 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:1881s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:1316s;cuda3:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__wikics__s3407:500s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:99%260811-05:26:30 (UTC+8)
1074tasks total=14270 running=3 done=2572 failed=0 blocked=0 pending=11695 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:1700s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:1135s;cuda3:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__wikics__s3407:319s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:61%260811-05:23:29 (UTC+8)
1073tasks total=14270 running=3 done=2572 failed=0 blocked=0 pending=11695 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:1519s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:954s;cuda3:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__wikics__s3407:138s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:98%260811-05:20:28 (UTC+8)
1072tasks total=14270 running=3 done=2571 failed=0 blocked=0 pending=11696 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:1338s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:773s;cuda3:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__wikics__s3407:719s | GPU cuda1:4885/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:99%260811-05:17:27 (UTC+8)
1071tasks total=14270 running=3 done=2571 failed=0 blocked=0 pending=11696 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:1157s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:592s;cuda3:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__wikics__s3407:538s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:6469/24564MiB:55%260811-05:14:26 (UTC+8)
1070tasks total=14270 running=3 done=2571 failed=0 blocked=0 pending=11696 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:976s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:411s;cuda3:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__wikics__s3407:357s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:100%260811-05:11:25 (UTC+8)
1069tasks total=14270 running=3 done=2571 failed=0 blocked=0 pending=11696 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:795s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:230s;cuda3:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__wikics__s3407:176s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:6469/24564MiB:98%260811-05:08:24 (UTC+8)
1068tasks total=14270 running=3 done=2568 failed=0 blocked=0 pending=11699 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:614s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s999:49s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:2300s | GPU cuda1:4885/24564MiB:96%;cuda2:7835/24564MiB:0%;cuda3:4367/24564MiB:46%260811-05:05:23 (UTC+8)
1067tasks total=14270 running=3 done=2566 failed=0 blocked=0 pending=11701 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:433s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:3751s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:2119s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-05:02:22 (UTC+8)
1066tasks total=14270 running=3 done=2566 failed=0 blocked=0 pending=11701 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:252s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:3570s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:1938s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:96%260811-04:59:21 (UTC+8)
1065tasks total=14270 running=3 done=2566 failed=0 blocked=0 pending=11701 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s999:70s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:3389s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:1756s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:100%260811-04:56:20 (UTC+8)
1064tasks total=14270 running=3 done=2565 failed=0 blocked=0 pending=11702 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:2893s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:3208s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:1575s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-04:53:19 (UTC+8)
1063tasks total=14270 running=3 done=2565 failed=0 blocked=0 pending=11702 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:2712s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:3027s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:1394s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-04:50:17 (UTC+8)
1062tasks total=14270 running=3 done=2565 failed=0 blocked=0 pending=11702 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:2531s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:2846s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:1214s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7349/24564MiB:100%260811-04:47:17 (UTC+8)
1061tasks total=14270 running=3 done=2565 failed=0 blocked=0 pending=11702 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:2350s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:2665s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:1033s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:97%260811-04:44:16 (UTC+8)
1060tasks total=14270 running=3 done=2565 failed=0 blocked=0 pending=11702 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:2169s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:2484s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:852s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-04:41:15 (UTC+8)
1059tasks total=14270 running=3 done=2565 failed=0 blocked=0 pending=11702 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:1988s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:2303s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:671s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-04:38:14 (UTC+8)
1058tasks total=14270 running=3 done=2565 failed=0 blocked=0 pending=11702 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:1807s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:2122s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:490s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7349/24564MiB:100%260811-04:35:13 (UTC+8)
1057tasks total=14270 running=3 done=2565 failed=0 blocked=0 pending=11702 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:1626s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:1941s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:309s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:99%260811-04:32:12 (UTC+8)
1056tasks total=14270 running=3 done=2565 failed=0 blocked=0 pending=11702 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:1445s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:1761s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s3407:128s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-04:29:11 (UTC+8)
1055tasks total=14270 running=3 done=2564 failed=0 blocked=0 pending=11703 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:1264s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:1579s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:2244s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7349/24564MiB:100%260811-04:26:10 (UTC+8)
1054tasks total=14270 running=3 done=2564 failed=0 blocked=0 pending=11703 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:1082s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:1397s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:2062s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-04:23:09 (UTC+8)
1053tasks total=14270 running=3 done=2564 failed=0 blocked=0 pending=11703 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:901s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:1216s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:1881s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:98%260811-04:20:07 (UTC+8)
1052tasks total=14270 running=3 done=2564 failed=0 blocked=0 pending=11703 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:720s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:1035s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:1700s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7349/24564MiB:100%260811-04:17:06 (UTC+8)
1051tasks total=14270 running=3 done=2564 failed=0 blocked=0 pending=11703 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:539s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:855s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:1520s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:99%260811-04:14:05 (UTC+8)
1050tasks total=14270 running=3 done=2564 failed=0 blocked=0 pending=11703 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:358s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:673s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:1338s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-04:11:04 (UTC+8)
1049tasks total=14270 running=3 done=2564 failed=0 blocked=0 pending=11703 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s999:177s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:492s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:1157s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-04:08:03 (UTC+8)
1048tasks total=14270 running=3 done=2562 failed=0 blocked=0 pending=11705 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:2760s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:311s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:976s | GPU cuda1:2463/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:96%260811-04:05:02 (UTC+8)
1047tasks total=14270 running=3 done=2562 failed=0 blocked=0 pending=11705 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:2579s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s999:130s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:795s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-04:02:01 (UTC+8)
1046tasks total=14270 running=3 done=2561 failed=0 blocked=0 pending=11706 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:2398s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:3825s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:615s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-03:59:00 (UTC+8)
1045tasks total=14270 running=3 done=2561 failed=0 blocked=0 pending=11706 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:2216s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:3644s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:433s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-03:55:59 (UTC+8)
1044tasks total=14270 running=3 done=2561 failed=0 blocked=0 pending=11706 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:2035s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:3463s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:252s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7349/24564MiB:100%260811-03:52:58 (UTC+8)
1043tasks total=14270 running=3 done=2561 failed=0 blocked=0 pending=11706 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:1855s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:3282s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s3407:71s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-03:49:57 (UTC+8)
1042tasks total=14270 running=3 done=2558 failed=0 blocked=0 pending=11709 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:1674s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:3101s;cuda3:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s3407:1224s | GPU cuda1:4625/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:99%260811-03:46:56 (UTC+8)
1041tasks total=14270 running=3 done=2558 failed=0 blocked=0 pending=11709 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:1493s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:2921s;cuda3:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s3407:1043s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:100%260811-03:43:55 (UTC+8)
1040tasks total=14270 running=3 done=2558 failed=0 blocked=0 pending=11709 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:1310s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:2738s;cuda3:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s3407:861s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:100%260811-03:40:54 (UTC+8)
1039tasks total=14270 running=3 done=2558 failed=0 blocked=0 pending=11709 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:1129s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:2557s;cuda3:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s3407:680s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:98%260811-03:37:52 (UTC+8)
1038tasks total=14270 running=3 done=2558 failed=0 blocked=0 pending=11709 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:949s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:2376s;cuda3:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s3407:499s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:100%260811-03:34:51 (UTC+8)
1037tasks total=14270 running=3 done=2558 failed=0 blocked=0 pending=11709 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:768s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:2195s;cuda3:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s3407:318s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260811-03:31:50 (UTC+8)
1036tasks total=14270 running=3 done=2558 failed=0 blocked=0 pending=11709 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:587s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:2014s;cuda3:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s3407:137s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:100%260811-03:28:49 (UTC+8)
1035tasks total=14270 running=3 done=2557 failed=0 blocked=0 pending=11710 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:405s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:1833s;cuda3:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s3407:1282s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:100%260811-03:25:48 (UTC+8)
1034tasks total=14270 running=3 done=2557 failed=0 blocked=0 pending=11710 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:224s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:1652s;cuda3:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s3407:1101s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260811-03:22:46 (UTC+8)
1033tasks total=14270 running=3 done=2557 failed=0 blocked=0 pending=11710 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s999:43s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:1471s;cuda3:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s3407:921s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:99%260811-03:19:46 (UTC+8)
1032tasks total=14270 running=3 done=2556 failed=0 blocked=0 pending=11711 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:2612s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:1290s;cuda3:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s3407:740s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:6789/24564MiB:98%260811-03:16:45 (UTC+8)
1031tasks total=14270 running=3 done=2556 failed=0 blocked=0 pending=11711 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:2431s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:1109s;cuda3:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s3407:558s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260811-03:13:44 (UTC+8)
1030tasks total=14270 running=3 done=2556 failed=0 blocked=0 pending=11711 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:2250s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:928s;cuda3:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s3407:378s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260811-03:10:43 (UTC+8)
1029tasks total=14270 running=3 done=2556 failed=0 blocked=0 pending=11711 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:2069s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:747s;cuda3:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s3407:197s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260811-03:07:42 (UTC+8)
1028tasks total=14270 running=3 done=2556 failed=0 blocked=0 pending=11711 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:1888s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:566s;cuda3:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s3407:16s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:6453/24564MiB:65%260811-03:04:41 (UTC+8)
1027tasks total=14270 running=3 done=2553 failed=0 blocked=0 pending=11714 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:1707s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:385s;cuda3:trace_context:trace_bank__base__Random-256-b__llama31_4b__wikics__s3407:767s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260811-03:01:40 (UTC+8)
1026tasks total=14270 running=3 done=2553 failed=0 blocked=0 pending=11714 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:1525s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:203s;cuda3:trace_context:trace_bank__base__Random-256-b__llama31_4b__wikics__s3407:586s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260811-02:58:39 (UTC+8)
1025tasks total=14270 running=3 done=2553 failed=0 blocked=0 pending=11714 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:1344s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s999:22s;cuda3:trace_context:trace_bank__base__Random-256-b__llama31_4b__wikics__s3407:405s | GPU cuda1:4625/24564MiB:0%;cuda2:7835/24564MiB:2%;cuda3:6529/24564MiB:100%260811-02:55:37 (UTC+8)
1024tasks total=14270 running=3 done=2551 failed=0 blocked=0 pending=11716 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:1163s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:3390s;cuda3:trace_context:trace_bank__base__Random-256-b__llama31_4b__wikics__s3407:223s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:97%260811-02:52:36 (UTC+8)
1023tasks total=14270 running=3 done=2551 failed=0 blocked=0 pending=11716 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:982s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:3209s;cuda3:trace_context:trace_bank__base__Random-256-b__llama31_4b__wikics__s3407:42s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:97%260811-02:49:35 (UTC+8)
1022tasks total=14270 running=3 done=2550 failed=0 blocked=0 pending=11717 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:802s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:3029s;cuda3:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s3407:791s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-02:46:34 (UTC+8)
1021tasks total=14270 running=3 done=2550 failed=0 blocked=0 pending=11717 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:620s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:2847s;cuda3:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s3407:610s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-02:43:33 (UTC+8)
1020tasks total=14270 running=3 done=2550 failed=0 blocked=0 pending=11717 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:439s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:2666s;cuda3:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s3407:429s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:94%;cuda3:6529/24564MiB:99%260811-02:40:32 (UTC+8)
1019tasks total=14270 running=3 done=2550 failed=0 blocked=0 pending=11717 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:258s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:2485s;cuda3:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s3407:248s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-02:37:31 (UTC+8)
1018tasks total=14270 running=3 done=2550 failed=0 blocked=0 pending=11717 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s999:77s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:2304s;cuda3:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s3407:67s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260811-02:34:30 (UTC+8)
1017tasks total=14270 running=3 done=2545 failed=0 blocked=0 pending=11722 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:2564s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:2123s;cuda3:trace_context:trace_bank__base__Random-128-b__llama31_4b__wikics__s3407:620s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:99%260811-02:31:29 (UTC+8)
1016tasks total=14270 running=3 done=2545 failed=0 blocked=0 pending=11722 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:2383s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:1942s;cuda3:trace_context:trace_bank__base__Random-128-b__llama31_4b__wikics__s3407:439s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:99%260811-02:28:28 (UTC+8)
1015tasks total=14270 running=3 done=2545 failed=0 blocked=0 pending=11722 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:2202s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:1761s;cuda3:trace_context:trace_bank__base__Random-128-b__llama31_4b__wikics__s3407:258s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:99%260811-02:25:26 (UTC+8)
1014tasks total=14270 running=3 done=2545 failed=0 blocked=0 pending=11722 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:2021s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:1580s;cuda3:trace_context:trace_bank__base__Random-128-b__llama31_4b__wikics__s3407:77s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:98%260811-02:22:26 (UTC+8)
1013tasks total=14270 running=3 done=2544 failed=0 blocked=0 pending=11723 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:1840s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:1399s;cuda3:trace_context:trace_bank__base__Random-128-a__llama31_4b__wikics__s3407:626s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:98%260811-02:19:25 (UTC+8)
1012tasks total=14270 running=3 done=2544 failed=0 blocked=0 pending=11723 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:1660s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:1218s;cuda3:trace_context:trace_bank__base__Random-128-a__llama31_4b__wikics__s3407:445s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:68%260811-02:16:24 (UTC+8)
1011tasks total=14270 running=3 done=2544 failed=0 blocked=0 pending=11723 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:1478s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:1037s;cuda3:trace_context:trace_bank__base__Random-128-a__llama31_4b__wikics__s3407:264s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:98%260811-02:13:23 (UTC+8)
1010tasks total=14270 running=3 done=2544 failed=0 blocked=0 pending=11723 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:1298s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:856s;cuda3:trace_context:trace_bank__base__Random-128-a__llama31_4b__wikics__s3407:83s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:99%260811-02:10:22 (UTC+8)
1009tasks total=14270 running=3 done=2541 failed=0 blocked=0 pending=11726 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:1116s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:675s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:2195s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7349/24564MiB:96%260811-02:07:21 (UTC+8)
1008tasks total=14270 running=3 done=2541 failed=0 blocked=0 pending=11726 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:935s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:494s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:2014s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-02:04:20 (UTC+8)
1007tasks total=14270 running=3 done=2541 failed=0 blocked=0 pending=11726 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:754s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:313s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:1833s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-02:01:19 (UTC+8)
1006tasks total=14270 running=3 done=2541 failed=0 blocked=0 pending=11726 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:573s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s999:132s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:1652s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7349/24564MiB:100%260811-01:58:18 (UTC+8)
1005tasks total=14270 running=3 done=2540 failed=0 blocked=0 pending=11727 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:392s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:3427s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:1471s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:55:17 (UTC+8)
1004tasks total=14270 running=3 done=2540 failed=0 blocked=0 pending=11727 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:212s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:3246s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:1290s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:52:16 (UTC+8)
1003tasks total=14270 running=3 done=2540 failed=0 blocked=0 pending=11727 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s999:31s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:3065s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:1110s | GPU cuda1:4549/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:49:15 (UTC+8)
1002tasks total=14270 running=3 done=2539 failed=0 blocked=0 pending=11728 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:2457s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:2884s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:928s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:100%260811-01:46:14 (UTC+8)
1001tasks total=14270 running=3 done=2539 failed=0 blocked=0 pending=11728 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:2276s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:2703s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:747s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:96%260811-01:43:13 (UTC+8)
1000tasks total=14270 running=3 done=2539 failed=0 blocked=0 pending=11728 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:2095s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:2522s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:566s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:98%260811-01:40:12 (UTC+8)
999tasks total=14270 running=3 done=2539 failed=0 blocked=0 pending=11728 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:1914s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:2341s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:386s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:37:11 (UTC+8)
998tasks total=14270 running=3 done=2539 failed=0 blocked=0 pending=11728 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:1733s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:2160s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:205s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:100%260811-01:34:10 (UTC+8)
997tasks total=14270 running=3 done=2539 failed=0 blocked=0 pending=11728 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:1552s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:1979s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s3407:23s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:98%;cuda3:7349/24564MiB:100%260811-01:31:09 (UTC+8)
996tasks total=14270 running=3 done=2538 failed=0 blocked=0 pending=11729 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:1371s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:1798s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:2137s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:28:08 (UTC+8)
995tasks total=14270 running=3 done=2538 failed=0 blocked=0 pending=11729 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:1190s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:1617s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:1957s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:25:07 (UTC+8)
994tasks total=14270 running=3 done=2538 failed=0 blocked=0 pending=11729 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:1009s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:1436s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:1776s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:22:06 (UTC+8)
993tasks total=14270 running=3 done=2538 failed=0 blocked=0 pending=11729 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:828s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:1255s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:1595s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:19:06 (UTC+8)
992tasks total=14270 running=3 done=2538 failed=0 blocked=0 pending=11729 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:647s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:1074s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:1414s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:7349/24564MiB:100%260811-01:16:05 (UTC+8)
991tasks total=14270 running=3 done=2538 failed=0 blocked=0 pending=11729 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:466s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:894s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:1233s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:7349/24564MiB:100%260811-01:13:04 (UTC+8)
990tasks total=14270 running=3 done=2538 failed=0 blocked=0 pending=11729 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:286s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:713s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:1052s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:10:03 (UTC+8)
989tasks total=14270 running=3 done=2538 failed=0 blocked=0 pending=11729 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s999:105s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:532s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:871s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:07:02 (UTC+8)
988tasks total=14270 running=3 done=2536 failed=0 blocked=0 pending=11731 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s999:1495s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:351s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:690s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-01:04:01 (UTC+8)
987tasks total=14270 running=3 done=2536 failed=0 blocked=0 pending=11731 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s999:1315s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s999:170s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:509s | GPU cuda1:5465/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:7349/24564MiB:100%260811-01:01:00 (UTC+8)
986tasks total=14270 running=3 done=2534 failed=0 blocked=0 pending=11733 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s999:1133s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:3270s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:328s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-00:57:59 (UTC+8)
985tasks total=14270 running=3 done=2534 failed=0 blocked=0 pending=11733 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s999:952s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:3089s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s3407:147s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7349/24564MiB:100%260811-00:54:58 (UTC+8)
984tasks total=14270 running=3 done=2531 failed=0 blocked=0 pending=11736 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s999:770s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:2907s;cuda3:trace_context:trace_bank__base__Full-256-b__llama31_4b__wikics__s3407:896s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:41%;cuda3:6529/24564MiB:100%260811-00:51:57 (UTC+8)
983tasks total=14270 running=3 done=2531 failed=0 blocked=0 pending=11736 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s999:589s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:2726s;cuda3:trace_context:trace_bank__base__Full-256-b__llama31_4b__wikics__s3407:715s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:6529/24564MiB:98%260811-00:48:55 (UTC+8)
982tasks total=14270 running=3 done=2531 failed=0 blocked=0 pending=11736 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s999:408s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:2545s;cuda3:trace_context:trace_bank__base__Full-256-b__llama31_4b__wikics__s3407:534s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:98%260811-00:45:54 (UTC+8)
981tasks total=14270 running=3 done=2531 failed=0 blocked=0 pending=11736 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s999:228s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:2364s;cuda3:trace_context:trace_bank__base__Full-256-b__llama31_4b__wikics__s3407:353s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:6529/24564MiB:98%260811-00:42:53 (UTC+8)
980tasks total=14270 running=3 done=2531 failed=0 blocked=0 pending=11736 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s999:47s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:2183s;cuda3:trace_context:trace_bank__base__Full-256-b__llama31_4b__wikics__s3407:172s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:6529/24564MiB:99%260811-00:39:52 (UTC+8)
979tasks total=14270 running=3 done=2529 failed=0 blocked=0 pending=11738 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s999:1436s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:2002s;cuda3:trace_context:trace_bank__base__Full-256-a__llama31_4b__wikics__s3407:922s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260811-00:36:51 (UTC+8)
978tasks total=14270 running=3 done=2529 failed=0 blocked=0 pending=11738 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s999:1254s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:1821s;cuda3:trace_context:trace_bank__base__Full-256-a__llama31_4b__wikics__s3407:741s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:99%260811-00:33:50 (UTC+8)
977tasks total=14270 running=3 done=2529 failed=0 blocked=0 pending=11738 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s999:1073s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:1640s;cuda3:trace_context:trace_bank__base__Full-256-a__llama31_4b__wikics__s3407:560s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-00:30:49 (UTC+8)
976tasks total=14270 running=3 done=2529 failed=0 blocked=0 pending=11738 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s999:892s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:1459s;cuda3:trace_context:trace_bank__base__Full-256-a__llama31_4b__wikics__s3407:379s | GPU cuda1:5465/24564MiB:93%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260811-00:27:48 (UTC+8)
975tasks total=14270 running=3 done=2529 failed=0 blocked=0 pending=11738 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s999:712s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:1278s;cuda3:trace_context:trace_bank__base__Full-256-a__llama31_4b__wikics__s3407:198s | GPU cuda1:5465/24564MiB:93%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260811-00:24:47 (UTC+8)
974tasks total=14270 running=3 done=2529 failed=0 blocked=0 pending=11738 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s999:529s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:1096s;cuda3:trace_context:trace_bank__base__Full-256-a__llama31_4b__wikics__s3407:16s | GPU cuda1:5465/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:6453/24564MiB:69%260811-00:21:46 (UTC+8)
973tasks total=14270 running=3 done=2526 failed=0 blocked=0 pending=11741 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s999:347s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:913s;cuda3:trace_context:trace_bank__base__Full-128-b__llama31_4b__wikics__s3407:567s | GPU cuda1:5465/24564MiB:94%;cuda2:7851/24564MiB:95%;cuda3:6469/24564MiB:99%260811-00:18:44 (UTC+8)
972tasks total=14270 running=3 done=2526 failed=0 blocked=0 pending=11741 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s999:165s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:731s;cuda3:trace_context:trace_bank__base__Full-128-b__llama31_4b__wikics__s3407:385s | GPU cuda1:5465/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:99%260811-00:15:41 (UTC+8)
971tasks total=14270 running=3 done=2524 failed=0 blocked=0 pending=11743 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s999:911s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:550s;cuda3:trace_context:trace_bank__base__Full-128-b__llama31_4b__wikics__s3407:204s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:13%;cuda3:6469/24564MiB:100%260811-00:12:39 (UTC+8)
970tasks total=14270 running=3 done=2524 failed=0 blocked=0 pending=11743 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s999:730s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:370s;cuda3:trace_context:trace_bank__base__Full-128-b__llama31_4b__wikics__s3407:23s | GPU cuda1:4885/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:99%260811-00:09:39 (UTC+8)
969tasks total=14270 running=3 done=2523 failed=0 blocked=0 pending=11744 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s999:550s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:189s;cuda3:trace_context:trace_bank__base__Full-128-a__llama31_4b__wikics__s3407:572s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:97%260811-00:06:38 (UTC+8)
968tasks total=14270 running=3 done=2523 failed=0 blocked=0 pending=11744 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s999:366s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s999:5s;cuda3:trace_context:trace_bank__base__Full-128-a__llama31_4b__wikics__s3407:388s | GPU cuda1:4885/24564MiB:3%;cuda2:401/24564MiB:0%;cuda3:6469/24564MiB:99%260811-00:03:37 (UTC+8)
967tasks total=14270 running=3 done=2522 failed=0 blocked=0 pending=11745 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s999:185s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:3144s;cuda3:trace_context:trace_bank__base__Full-128-a__llama31_4b__wikics__s3407:208s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:100%260811-00:00:33 (UTC+8)
966tasks total=14270 running=3 done=2522 failed=0 blocked=0 pending=11745 | results done=869/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama31_4b__wikics__s999:4s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:2963s;cuda3:trace_context:trace_bank__base__Full-128-a__llama31_4b__wikics__s3407:27s | GPU cuda1:401/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:6469/24564MiB:98%260810-23:57:32 (UTC+8)
965tasks total=14270 running=3 done=2515 failed=0 blocked=0 pending=11752 | results done=864/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s999:748s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:2782s;cuda3:fixed_downstream:run__T2_FixedSemantic256__llama31_4b__wikics__s3407__gcn2:36s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:3457/24564MiB:99%260810-23:54:32 (UTC+8)
964tasks total=14270 running=3 done=2512 failed=0 blocked=0 pending=11755 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s999:567s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:2601s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__wikics__s3407:842s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260810-23:51:31 (UTC+8)
963tasks total=14270 running=3 done=2512 failed=0 blocked=0 pending=11755 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s999:386s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:2420s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__wikics__s3407:661s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:99%260810-23:48:30 (UTC+8)
962tasks total=14270 running=3 done=2512 failed=0 blocked=0 pending=11755 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s999:205s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:2239s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__wikics__s3407:480s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260810-23:45:29 (UTC+8)
961tasks total=14270 running=3 done=2512 failed=0 blocked=0 pending=11755 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s999:24s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:2058s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__wikics__s3407:299s | GPU cuda1:4885/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260810-23:42:28 (UTC+8)
960tasks total=14270 running=3 done=2510 failed=0 blocked=0 pending=11757 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama31_4b__wikics__s999:372s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:1877s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__wikics__s3407:118s | GPU cuda1:4565/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260810-23:39:27 (UTC+8)
959tasks total=14270 running=3 done=2509 failed=0 blocked=0 pending=11758 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama31_4b__wikics__s999:191s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:1696s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__wikics__s3407:897s | GPU cuda1:4565/24564MiB:93%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260810-23:36:26 (UTC+8)
958tasks total=14270 running=3 done=2509 failed=0 blocked=0 pending=11758 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama31_4b__wikics__s999:10s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:1515s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__wikics__s3407:716s | GPU cuda1:4433/24564MiB:21%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260810-23:33:25 (UTC+8)
957tasks total=14270 running=3 done=2508 failed=0 blocked=0 pending=11759 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama31_4b__wikics__s999:356s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:1335s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__wikics__s3407:535s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260810-23:30:24 (UTC+8)
956tasks total=14270 running=3 done=2508 failed=0 blocked=0 pending=11759 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama31_4b__wikics__s999:175s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:1153s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__wikics__s3407:354s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:100%260810-23:27:23 (UTC+8)
955tasks total=14270 running=3 done=2506 failed=0 blocked=0 pending=11761 | results done=863/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s999:1540s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:972s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__wikics__s3407:173s | GPU cuda1:5465/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:6529/24564MiB:98%260810-23:24:22 (UTC+8)
954tasks total=14270 running=3 done=2504 failed=0 blocked=0 pending=11763 | results done=862/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s999:1359s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:791s;cuda3:fixed_downstream:run__T1_FixedFull512__llama31_4b__wikics__s3407__sage3:35s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:3345/24564MiB:99%260810-23:21:21 (UTC+8)
953tasks total=14270 running=3 done=2499 failed=0 blocked=0 pending=11768 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s999:1178s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:610s;cuda3:fixed_downstream:run__T1_FixedFull512__llama31_4b__wikics__s3407__gcn1:22s | GPU cuda1:5445/24564MiB:99%;cuda2:7851/24564MiB:0%;cuda3:3413/24564MiB:99%260810-23:18:20 (UTC+8)
952tasks total=14270 running=3 done=2497 failed=0 blocked=0 pending=11770 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s999:997s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:429s;cuda3:trace_context:trace_bank__base__Full-512-b__llama31_4b__wikics__s3407:1170s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260810-23:15:19 (UTC+8)
951tasks total=14270 running=3 done=2497 failed=0 blocked=0 pending=11770 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s999:815s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:247s;cuda3:trace_context:trace_bank__base__Full-512-b__llama31_4b__wikics__s3407:987s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:97%260810-23:12:18 (UTC+8)
950tasks total=14270 running=3 done=2497 failed=0 blocked=0 pending=11770 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s999:631s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s999:63s;cuda3:trace_context:trace_bank__base__Full-512-b__llama31_4b__wikics__s3407:804s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:6789/24564MiB:99%260810-23:09:15 (UTC+8)
949tasks total=14270 running=3 done=2495 failed=0 blocked=0 pending=11772 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s999:450s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:2360s;cuda3:trace_context:trace_bank__base__Full-512-b__llama31_4b__wikics__s3407:622s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:97%260810-23:06:12 (UTC+8)
948tasks total=14270 running=3 done=2495 failed=0 blocked=0 pending=11772 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s999:269s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:2179s;cuda3:trace_context:trace_bank__base__Full-512-b__llama31_4b__wikics__s3407:441s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:100%260810-23:03:11 (UTC+8)
947tasks total=14270 running=3 done=2495 failed=0 blocked=0 pending=11772 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s999:88s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:1998s;cuda3:trace_context:trace_bank__base__Full-512-b__llama31_4b__wikics__s3407:260s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:100%260810-23:00:10 (UTC+8)
946tasks total=14270 running=3 done=2494 failed=0 blocked=0 pending=11773 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s999:1452s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:1817s;cuda3:trace_context:trace_bank__base__Full-512-b__llama31_4b__wikics__s3407:80s | GPU cuda1:5465/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:6789/24564MiB:100%260810-22:57:09 (UTC+8)
945tasks total=14270 running=3 done=2493 failed=0 blocked=0 pending=11774 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s999:1269s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:1633s;cuda3:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s3407:1226s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:26%;cuda3:6789/24564MiB:100%260810-22:54:08 (UTC+8)
944tasks total=14270 running=3 done=2493 failed=0 blocked=0 pending=11774 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s999:1088s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:1453s;cuda3:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s3407:1045s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:100%260810-22:51:04 (UTC+8)
943tasks total=14270 running=3 done=2493 failed=0 blocked=0 pending=11774 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s999:907s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:1272s;cuda3:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s3407:864s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:99%260810-22:48:03 (UTC+8)
942tasks total=14270 running=3 done=2493 failed=0 blocked=0 pending=11774 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s999:725s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:1090s;cuda3:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s3407:682s | GPU cuda1:5445/24564MiB:93%;cuda2:7851/24564MiB:100%;cuda3:6789/24564MiB:99%260810-22:45:03 (UTC+8)
941tasks total=14270 running=3 done=2493 failed=0 blocked=0 pending=11774 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s999:544s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:909s;cuda3:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s3407:501s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:6789/24564MiB:100%260810-22:42:01 (UTC+8)
940tasks total=14270 running=3 done=2493 failed=0 blocked=0 pending=11774 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s999:363s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:728s;cuda3:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s3407:320s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:100%260810-22:39:00 (UTC+8)
939tasks total=14270 running=3 done=2493 failed=0 blocked=0 pending=11774 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s999:182s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:547s;cuda3:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s3407:139s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:6789/24564MiB:100%260810-22:35:59 (UTC+8)
938tasks total=14270 running=3 done=2492 failed=0 blocked=0 pending=11775 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s999:1s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:366s;cuda3:trace_context:trace_bank__base__Self__llama31_4b__wikics__s3407:555s | GPU cuda1:14/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:6469/24564MiB:99%260810-22:32:58 (UTC+8)
937tasks total=14270 running=3 done=2490 failed=0 blocked=0 pending=11777 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s999:721s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:185s;cuda3:trace_context:trace_bank__base__Self__llama31_4b__wikics__s3407:374s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:6469/24564MiB:99%260810-22:29:56 (UTC+8)
936tasks total=14270 running=3 done=2490 failed=0 blocked=0 pending=11777 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s999:540s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s999:4s;cuda3:trace_context:trace_bank__base__Self__llama31_4b__wikics__s3407:193s | GPU cuda1:4885/24564MiB:96%;cuda2:401/24564MiB:0%;cuda3:6469/24564MiB:99%260810-22:26:55 (UTC+8)
935tasks total=14270 running=3 done=2489 failed=0 blocked=0 pending=11778 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s999:358s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:2320s;cuda3:trace_context:trace_bank__base__Self__llama31_4b__wikics__s3407:11s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:6341/24564MiB:42%260810-22:23:55 (UTC+8)
934tasks total=14270 running=3 done=2488 failed=0 blocked=0 pending=11779 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s999:177s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:2139s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:2647s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:96%;cuda3:9873/24564MiB:100%260810-22:20:53 (UTC+8)
933tasks total=14270 running=3 done=2487 failed=0 blocked=0 pending=11780 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s999:896s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:1958s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:2466s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:9873/24564MiB:100%260810-22:17:52 (UTC+8)
932tasks total=14270 running=3 done=2487 failed=0 blocked=0 pending=11780 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s999:715s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:1777s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:2285s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:7969/24564MiB:94%260810-22:14:51 (UTC+8)
931tasks total=14270 running=3 done=2487 failed=0 blocked=0 pending=11780 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s999:533s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:1595s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:2103s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:7969/24564MiB:99%260810-22:11:50 (UTC+8)
930tasks total=14270 running=3 done=2487 failed=0 blocked=0 pending=11780 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s999:352s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:1414s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:1922s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:7969/24564MiB:95%260810-22:08:48 (UTC+8)
929tasks total=14270 running=3 done=2487 failed=0 blocked=0 pending=11780 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s999:171s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:1233s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:1742s | GPU cuda1:4885/24564MiB:95%;cuda2:7851/24564MiB:96%;cuda3:7969/24564MiB:99%260810-22:05:47 (UTC+8)
928tasks total=14270 running=3 done=2485 failed=0 blocked=0 pending=11782 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama31_4b__wikics__s999:626s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:1052s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:1560s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7969/24564MiB:99%260810-22:02:46 (UTC+8)
927tasks total=14270 running=3 done=2485 failed=0 blocked=0 pending=11782 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama31_4b__wikics__s999:445s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:870s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:1379s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:7969/24564MiB:100%260810-21:59:45 (UTC+8)
926tasks total=14270 running=3 done=2485 failed=0 blocked=0 pending=11782 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama31_4b__wikics__s999:264s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:690s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:1198s | GPU cuda1:4625/24564MiB:98%;cuda2:7851/24564MiB:95%;cuda3:7969/24564MiB:99%260810-21:56:44 (UTC+8)
925tasks total=14270 running=3 done=2485 failed=0 blocked=0 pending=11782 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama31_4b__wikics__s999:81s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:507s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:1016s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:7969/24564MiB:99%260810-21:53:43 (UTC+8)
924tasks total=14270 running=3 done=2484 failed=0 blocked=0 pending=11783 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s999:534s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:326s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:835s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7969/24564MiB:100%260810-21:50:40 (UTC+8)
923tasks total=14270 running=3 done=2484 failed=0 blocked=0 pending=11783 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s999:353s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s999:145s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:654s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7969/24564MiB:99%260810-21:47:39 (UTC+8)
922tasks total=14270 running=3 done=2482 failed=0 blocked=0 pending=11785 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s999:172s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s999:1508s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:473s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:7967/24564MiB:95%260810-21:44:39 (UTC+8)
921tasks total=14270 running=3 done=2480 failed=0 blocked=0 pending=11787 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama31_4b__wikics__s999:494s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s999:1328s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:292s | GPU cuda1:4565/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:7967/24564MiB:100%260810-21:41:37 (UTC+8)
920tasks total=14270 running=3 done=2480 failed=0 blocked=0 pending=11787 | results done=857/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama31_4b__wikics__s999:313s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s999:1147s;cuda3:sft:sft__trace__llama31_4b__wikics__s3407:111s | GPU cuda1:4565/24564MiB:57%;cuda2:7851/24564MiB:99%;cuda3:7967/24564MiB:100%260810-21:38:37 (UTC+8)
919tasks total=14270 running=3 done=2476 failed=0 blocked=0 pending=11791 | results done=854/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama31_4b__wikics__s999:131s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s999:965s;cuda3:external_downstream:run__LinguGKD__llama3_8b__wikics__s999__sage1:15s | GPU cuda1:4565/24564MiB:95%;cuda2:7851/24564MiB:96%;cuda3:1421/24564MiB:22%260810-21:35:36 (UTC+8)
918tasks total=14270 running=3 done=2470 failed=0 blocked=0 pending=11797 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama31_4b__wikics__s999:454s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s999:783s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:2251s | GPU cuda1:4565/24564MiB:93%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-21:32:34 (UTC+8)
917tasks total=14270 running=3 done=2470 failed=0 blocked=0 pending=11797 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama31_4b__wikics__s999:273s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s999:602s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:2070s | GPU cuda1:4565/24564MiB:97%;cuda2:7861/24564MiB:100%;cuda3:7851/24564MiB:0%260810-21:29:32 (UTC+8)
916tasks total=14270 running=3 done=2470 failed=0 blocked=0 pending=11797 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama31_4b__wikics__s999:92s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s999:421s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1889s | GPU cuda1:4565/24564MiB:97%;cuda2:7861/24564MiB:97%;cuda3:7851/24564MiB:96%260810-21:26:31 (UTC+8)
915tasks total=14270 running=3 done=2468 failed=0 blocked=0 pending=11799 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s999:1462s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s999:240s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1708s | GPU cuda1:5465/24564MiB:94%;cuda2:7861/24564MiB:97%;cuda3:7851/24564MiB:100%260810-21:23:30 (UTC+8)
914tasks total=14270 running=3 done=2468 failed=0 blocked=0 pending=11799 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s999:1279s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s999:57s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1526s | GPU cuda1:5465/24564MiB:100%;cuda2:7861/24564MiB:97%;cuda3:7851/24564MiB:96%260810-21:20:29 (UTC+8)
913tasks total=14270 running=3 done=2467 failed=0 blocked=0 pending=11800 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s999:1098s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s999:1406s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1345s | GPU cuda1:5465/24564MiB:94%;cuda2:7861/24564MiB:0%;cuda3:7851/24564MiB:96%260810-21:17:26 (UTC+8)
912tasks total=14270 running=3 done=2467 failed=0 blocked=0 pending=11800 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s999:917s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s999:1225s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:1164s | GPU cuda1:5465/24564MiB:94%;cuda2:7861/24564MiB:96%;cuda3:7851/24564MiB:100%260810-21:14:25 (UTC+8)
911tasks total=14270 running=3 done=2467 failed=0 blocked=0 pending=11800 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s999:736s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s999:1044s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:983s | GPU cuda1:5465/24564MiB:100%;cuda2:7861/24564MiB:100%;cuda3:7851/24564MiB:96%260810-21:11:24 (UTC+8)
910tasks total=14270 running=3 done=2467 failed=0 blocked=0 pending=11800 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s999:555s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s999:863s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:802s | GPU cuda1:5465/24564MiB:95%;cuda2:7861/24564MiB:97%;cuda3:7851/24564MiB:96%260810-21:08:23 (UTC+8)
909tasks total=14270 running=3 done=2467 failed=0 blocked=0 pending=11800 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s999:374s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s999:682s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:621s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:97%260810-21:05:22 (UTC+8)
908tasks total=14270 running=3 done=2467 failed=0 blocked=0 pending=11800 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s999:5s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s999:314s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:252s | GPU cuda1:401/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:66%260810-20:59:14 (UTC+8)
907tasks total=14270 running=3 done=2466 failed=0 blocked=0 pending=11801 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s999:1371s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s999:133s;cuda3:lingu_bank:lingu_bank__llama3_8b__wikics__s999:71s | GPU cuda1:5465/24564MiB:6%;cuda2:7851/24564MiB:93%;cuda3:7851/24564MiB:100%260810-20:56:13 (UTC+8)
906tasks total=14270 running=3 done=2463 failed=0 blocked=0 pending=11804 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s999:812s;cuda2:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s999:503s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:6020s | GPU cuda1:5465/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:11839/24564MiB:100%260810-20:46:54 (UTC+8)
905tasks total=14270 running=3 done=2463 failed=0 blocked=0 pending=11804 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s999:631s;cuda2:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s999:322s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:5839s | GPU cuda1:5465/24564MiB:89%;cuda2:7851/24564MiB:98%;cuda3:11839/24564MiB:100%260810-20:43:53 (UTC+8)
904tasks total=14270 running=3 done=2463 failed=0 blocked=0 pending=11804 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s999:450s;cuda2:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s999:141s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:5658s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:95%;cuda3:11839/24564MiB:100%260810-20:40:52 (UTC+8)
903tasks total=14270 running=3 done=2462 failed=0 blocked=0 pending=11805 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s999:78s;cuda2:trace_context:trace_bank__n1__Semantic-128-a__llama3_8b__wikics__s999:691s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:5287s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:97%;cuda3:11839/24564MiB:99%260810-20:34:43 (UTC+8)
902tasks total=14270 running=3 done=2457 failed=0 blocked=0 pending=11810 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-a__llama31_4b__wikics__s999:597s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s999:2343s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:4537s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:11839/24564MiB:100%260810-20:22:11 (UTC+8)
901tasks total=14270 running=3 done=2455 failed=0 blocked=0 pending=11812 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-b__llama31_4b__wikics__s999:354s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s999:1597s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:3791s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:11839/24564MiB:100%260810-20:09:44 (UTC+8)
900tasks total=14270 running=3 done=2455 failed=0 blocked=0 pending=11812 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-b__llama31_4b__wikics__s999:172s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s999:1415s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:3609s | GPU cuda1:4565/24564MiB:52%;cuda2:7851/24564MiB:94%;cuda3:11839/24564MiB:100%260810-20:06:42 (UTC+8)
899tasks total=14270 running=3 done=2454 failed=0 blocked=0 pending=11813 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s999:304s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s999:1046s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:3240s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:11839/24564MiB:100%260810-20:00:34 (UTC+8)
898tasks total=14270 running=3 done=2452 failed=0 blocked=0 pending=11815 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s999:597s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s999:678s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:2872s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:11839/24564MiB:99%260810-19:54:25 (UTC+8)
897tasks total=14270 running=3 done=2452 failed=0 blocked=0 pending=11815 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s999:416s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s999:497s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:2691s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:11839/24564MiB:100%260810-19:51:24 (UTC+8)
896tasks total=14270 running=3 done=2452 failed=0 blocked=0 pending=11815 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s999:47s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s999:128s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:2322s | GPU cuda1:4625/24564MiB:45%;cuda2:7851/24564MiB:100%;cuda3:11839/24564MiB:100%260810-19:45:15 (UTC+8)
895tasks total=14270 running=3 done=2450 failed=0 blocked=0 pending=11817 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s999:526s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:2341s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:2141s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:11839/24564MiB:100%260810-19:42:14 (UTC+8)
894tasks total=14270 running=3 done=2450 failed=0 blocked=0 pending=11817 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s999:345s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:2160s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:1960s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:11839/24564MiB:100%260810-19:39:13 (UTC+8)
893tasks total=14270 running=3 done=2450 failed=0 blocked=0 pending=11817 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s999:164s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:1979s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:1779s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:99%;cuda3:11839/24564MiB:100%260810-19:36:12 (UTC+8)
892tasks total=14270 running=3 done=2448 failed=0 blocked=0 pending=11819 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s999:697s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:1610s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:1410s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:11839/24564MiB:99%260810-19:30:04 (UTC+8)
891tasks total=14270 running=3 done=2448 failed=0 blocked=0 pending=11819 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s999:516s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:1429s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:1229s | GPU cuda1:4885/24564MiB:98%;cuda2:7851/24564MiB:95%;cuda3:11839/24564MiB:100%260810-19:27:03 (UTC+8)
890tasks total=14270 running=3 done=2448 failed=0 blocked=0 pending=11819 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s999:147s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:1060s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:860s | GPU cuda1:4885/24564MiB:99%;cuda2:7851/24564MiB:60%;cuda3:11839/24564MiB:100%260810-19:20:54 (UTC+8)
889tasks total=14270 running=3 done=2447 failed=0 blocked=0 pending=11820 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s999:863s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:878s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:678s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:11839/24564MiB:98%260810-19:17:53 (UTC+8)
888tasks total=14270 running=3 done=2447 failed=0 blocked=0 pending=11820 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s999:682s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:697s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:497s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:100%;cuda3:11839/24564MiB:100%260810-19:14:50 (UTC+8)
887tasks total=14270 running=3 done=2447 failed=0 blocked=0 pending=11820 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s999:502s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:516s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:316s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:0%;cuda3:11839/24564MiB:100%260810-19:11:50 (UTC+8)
886tasks total=14270 running=3 done=2447 failed=0 blocked=0 pending=11820 | results done=851/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s999:321s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:335s;cuda3:sft:sft__lingu__llama3_8b__wikics__s999:135s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:11665/24564MiB:99%260810-19:08:49 (UTC+8)
885tasks total=14270 running=3 done=2444 failed=0 blocked=0 pending=11823 | results done=850/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s999:138s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s999:153s;cuda3:trace_analysis:trace_analysis__n2__llama3_8b__wikics__s888:53s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:1631/24564MiB:8%260810-19:05:48 (UTC+8)
884tasks total=14270 running=3 done=2438 failed=0 blocked=0 pending=11829 | results done=849/5050 | RUNNING cuda1:trace_analysis:trace_analysis__n1__llama31_4b__wikics__s999:56s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s999:1474s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:2078s | GPU cuda1:1631/24564MiB:8%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:94%260810-19:02:45 (UTC+8)
883tasks total=14270 running=3 done=2436 failed=0 blocked=0 pending=11831 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s999:1322s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s999:1293s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:1898s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-18:59:44 (UTC+8)
882tasks total=14270 running=3 done=2436 failed=0 blocked=0 pending=11831 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s999:1141s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s999:1113s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:1717s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:96%260810-18:56:43 (UTC+8)
881tasks total=14270 running=3 done=2436 failed=0 blocked=0 pending=11831 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s999:960s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s999:931s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:1535s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:100%260810-18:53:42 (UTC+8)
880tasks total=14270 running=3 done=2436 failed=0 blocked=0 pending=11831 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s999:778s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s999:750s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:1354s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-18:50:41 (UTC+8)
879tasks total=14270 running=3 done=2436 failed=0 blocked=0 pending=11831 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s999:598s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s999:569s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:1173s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:96%260810-18:47:40 (UTC+8)
878tasks total=14270 running=3 done=2436 failed=0 blocked=0 pending=11831 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s999:417s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s999:388s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:993s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:98%260810-18:44:39 (UTC+8)
877tasks total=14270 running=3 done=2436 failed=0 blocked=0 pending=11831 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s999:236s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s999:207s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:812s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:29%260810-18:41:38 (UTC+8)
876tasks total=14270 running=3 done=2436 failed=0 blocked=0 pending=11831 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s999:53s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s999:25s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:629s | GPU cuda1:5445/24564MiB:95%;cuda2:7835/24564MiB:0%;cuda3:7851/24564MiB:96%260810-18:38:37 (UTC+8)
875tasks total=14270 running=3 done=2434 failed=0 blocked=0 pending=11833 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s999:1315s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s999:1339s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:445s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-18:35:35 (UTC+8)
874tasks total=14270 running=3 done=2434 failed=0 blocked=0 pending=11833 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s999:1135s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s999:1158s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:265s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:96%260810-18:32:31 (UTC+8)
873tasks total=14270 running=3 done=2434 failed=0 blocked=0 pending=11833 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s999:953s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s999:977s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s888:83s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-18:29:30 (UTC+8)
872tasks total=14270 running=3 done=2433 failed=0 blocked=0 pending=11834 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s999:773s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s999:796s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:2107s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:96%260810-18:26:29 (UTC+8)
871tasks total=14270 running=3 done=2433 failed=0 blocked=0 pending=11834 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s999:590s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s999:613s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:1925s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:98%;cuda3:7851/24564MiB:96%260810-18:23:28 (UTC+8)
870tasks total=14270 running=3 done=2433 failed=0 blocked=0 pending=11834 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s999:409s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s999:432s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:1744s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260810-18:20:26 (UTC+8)
869tasks total=14270 running=3 done=2433 failed=0 blocked=0 pending=11834 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s999:226s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s999:249s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:1561s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-18:17:25 (UTC+8)
868tasks total=14270 running=3 done=2433 failed=0 blocked=0 pending=11834 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s999:45s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s999:68s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:1380s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-18:14:22 (UTC+8)
867tasks total=14270 running=3 done=2429 failed=0 blocked=0 pending=11838 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s999:739s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s999:971s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:1199s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:19%;cuda3:7851/24564MiB:100%260810-18:11:21 (UTC+8)
866tasks total=14270 running=3 done=2429 failed=0 blocked=0 pending=11838 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s999:558s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s999:790s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:1018s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-18:08:20 (UTC+8)
865tasks total=14270 running=3 done=2429 failed=0 blocked=0 pending=11838 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s999:377s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s999:609s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:837s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-18:05:19 (UTC+8)
864tasks total=14270 running=3 done=2429 failed=0 blocked=0 pending=11838 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s999:196s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s999:428s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:656s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:96%260810-18:02:18 (UTC+8)
863tasks total=14270 running=3 done=2429 failed=0 blocked=0 pending=11838 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s999:15s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s999:247s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:475s | GPU cuda1:4549/24564MiB:0%;cuda2:7851/24564MiB:85%;cuda3:7851/24564MiB:96%260810-17:59:17 (UTC+8)
862tasks total=14270 running=3 done=2428 failed=0 blocked=0 pending=11839 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s999:709s;cuda2:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s999:66s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:294s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-17:56:16 (UTC+8)
861tasks total=14270 running=3 done=2427 failed=0 blocked=0 pending=11840 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s999:528s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s999:964s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s888:114s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:96%260810-17:53:15 (UTC+8)
860tasks total=14270 running=3 done=2425 failed=0 blocked=0 pending=11842 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s999:346s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s999:782s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s888:1305s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:100%260810-17:50:14 (UTC+8)
859tasks total=14270 running=3 done=2425 failed=0 blocked=0 pending=11842 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s999:166s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s999:602s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s888:1125s | GPU cuda1:4865/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-17:47:12 (UTC+8)
858tasks total=14270 running=3 done=2423 failed=0 blocked=0 pending=11844 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s999:566s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s999:421s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s888:944s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:7851/24564MiB:97%260810-17:44:12 (UTC+8)
857tasks total=14270 running=3 done=2423 failed=0 blocked=0 pending=11844 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s999:384s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s999:239s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s888:762s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:100%260810-17:41:11 (UTC+8)
856tasks total=14270 running=3 done=2423 failed=0 blocked=0 pending=11844 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s999:202s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s999:57s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s888:580s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:95%260810-17:38:09 (UTC+8)
855tasks total=14270 running=3 done=2421 failed=0 blocked=0 pending=11846 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s999:22s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s999:772s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s888:400s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:7851/24564MiB:97%260810-17:35:07 (UTC+8)
854tasks total=14270 running=3 done=2420 failed=0 blocked=0 pending=11847 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-a__llama31_4b__wikics__s999:418s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s999:590s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s888:218s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:98%;cuda3:7851/24564MiB:0%260810-17:32:07 (UTC+8)
853tasks total=14270 running=3 done=2420 failed=0 blocked=0 pending=11847 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-a__llama31_4b__wikics__s999:237s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s999:409s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s888:37s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:97%260810-17:29:05 (UTC+8)
852tasks total=14270 running=3 done=2419 failed=0 blocked=0 pending=11848 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-a__llama31_4b__wikics__s999:55s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s999:227s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s888:1226s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260810-17:26:04 (UTC+8)
851tasks total=14270 running=3 done=2417 failed=0 blocked=0 pending=11850 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-b__llama31_4b__wikics__s999:336s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s999:46s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s888:1045s | GPU cuda1:4565/24564MiB:95%;cuda2:7851/24564MiB:98%;cuda3:7851/24564MiB:97%260810-17:23:02 (UTC+8)
850tasks total=14270 running=3 done=2416 failed=0 blocked=0 pending=11851 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-b__llama31_4b__wikics__s999:155s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s999:757s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s888:864s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:97%260810-17:20:01 (UTC+8)
849tasks total=14270 running=3 done=2415 failed=0 blocked=0 pending=11852 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama31_4b__wikics__s999:432s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s999:576s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s888:683s | GPU cuda1:4565/24564MiB:95%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:95%260810-17:17:00 (UTC+8)
848tasks total=14270 running=3 done=2415 failed=0 blocked=0 pending=11852 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama31_4b__wikics__s999:250s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s999:394s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s888:501s | GPU cuda1:4565/24564MiB:57%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260810-17:13:58 (UTC+8)
847tasks total=14270 running=3 done=2415 failed=0 blocked=0 pending=11852 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama31_4b__wikics__s999:69s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s999:214s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s888:320s | GPU cuda1:4565/24564MiB:34%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:77%260810-17:10:57 (UTC+8)
846tasks total=14270 running=3 done=2413 failed=0 blocked=0 pending=11854 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:3881s;cuda2:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s999:32s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s888:138s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:7851/24564MiB:97%260810-17:07:56 (UTC+8)
845tasks total=14270 running=3 done=2409 failed=0 blocked=0 pending=11858 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:3700s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s999:2216s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s888:916s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:97%260810-17:04:54 (UTC+8)
844tasks total=14270 running=3 done=2409 failed=0 blocked=0 pending=11858 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:3519s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s999:2036s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s888:736s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:97%260810-17:01:53 (UTC+8)
843tasks total=14270 running=3 done=2409 failed=0 blocked=0 pending=11858 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:3338s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s999:1854s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s888:554s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:97%260810-16:58:52 (UTC+8)
842tasks total=14270 running=3 done=2409 failed=0 blocked=0 pending=11858 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:3155s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s999:1672s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s888:372s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:97%260810-16:55:51 (UTC+8)
841tasks total=14270 running=3 done=2409 failed=0 blocked=0 pending=11858 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:2975s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s999:1491s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s888:191s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:97%260810-16:52:49 (UTC+8)
840tasks total=14270 running=3 done=2409 failed=0 blocked=0 pending=11858 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:2794s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s999:1310s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s888:10s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7653/24564MiB:13%260810-16:49:48 (UTC+8)
839tasks total=14270 running=3 done=2408 failed=0 blocked=0 pending=11859 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:2612s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s999:1129s;cuda3:trace_context:trace_bank__n2__HopQuota-256-a__llama3_8b__wikics__s888:785s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:97%260810-16:46:47 (UTC+8)
838tasks total=14270 running=3 done=2408 failed=0 blocked=0 pending=11859 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:2429s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s999:946s;cuda3:trace_context:trace_bank__n2__HopQuota-256-a__llama3_8b__wikics__s888:602s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:97%260810-16:43:46 (UTC+8)
837tasks total=14270 running=3 done=2405 failed=0 blocked=0 pending=11862 | results done=849/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s999:1233s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s999:2114s;cuda3:trace_context:trace_bank__n2__HopQuota-128-b__llama3_8b__wikics__s888:216s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:98%260810-16:24:01 (UTC+8)
836tasks total=14270 running=3 done=2122 failed=0 blocked=0 pending=12145 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s999:555s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s999:1180s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s888:2117s | GPU cuda1:4905/24564MiB:97%;cuda2:7851/24564MiB:0%;cuda3:7851/24564MiB:100%260810-01:49:37 (UTC+8)
835tasks total=14270 running=3 done=2122 failed=0 blocked=0 pending=12145 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s999:1s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s999:626s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s888:1563s | GPU cuda1:14/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7851/24564MiB:96%260810-01:40:24 (UTC+8)
834tasks total=14270 running=3 done=2115 failed=0 blocked=0 pending=12152 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__wikics__s999:398s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s999:2282s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s888:824s | GPU cuda1:4605/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:100%260810-01:28:04 (UTC+8)
833tasks total=14270 running=3 done=2112 failed=0 blocked=0 pending=12155 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-b__llama31_4b__wikics__s999:111s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s999:1540s;cuda3:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s888:82s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:100%260810-01:15:42 (UTC+8)
832tasks total=14270 running=3 done=2109 failed=0 blocked=0 pending=12158 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__wikics__s999:3s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s999:981s;cuda3:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s888:573s | GPU cuda1:401/24564MiB:0%;cuda2:7851/24564MiB:56%;cuda3:7851/24564MiB:37%260810-01:06:23 (UTC+8)
831tasks total=14270 running=3 done=2106 failed=0 blocked=0 pending=12161 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s999:3514s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s999:800s;cuda3:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s888:392s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:95%260810-01:03:22 (UTC+8)
830tasks total=14270 running=3 done=2104 failed=0 blocked=0 pending=12163 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s999:2221s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s999:1894s;cuda3:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s888:147s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:97%260810-00:41:49 (UTC+8)
829tasks total=14270 running=3 done=2101 failed=0 blocked=0 pending=12166 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s999:1106s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s999:778s;cuda3:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s888:771s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:76%260810-00:23:14 (UTC+8)
828tasks total=14270 running=3 done=2101 failed=0 blocked=0 pending=12166 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s999:925s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s999:597s;cuda3:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s888:589s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:98%260810-00:20:13 (UTC+8)
827tasks total=14270 running=3 done=2101 failed=0 blocked=0 pending=12166 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s999:373s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s999:46s;cuda3:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s888:38s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:98%260810-00:11:01 (UTC+8)
826tasks total=14270 running=3 done=2096 failed=0 blocked=0 pending=12171 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s999:192s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s999:1373s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s888:932s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:95%260810-00:08:00 (UTC+8)
825tasks total=14270 running=3 done=2096 failed=0 blocked=0 pending=12171 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s999:11s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s999:1192s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s888:750s | GPU cuda1:4437/24564MiB:23%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260810-00:04:59 (UTC+8)
824tasks total=14270 running=3 done=2095 failed=0 blocked=0 pending=12172 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s999:3297s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s999:825s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s888:384s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260809-23:58:52 (UTC+8)
823tasks total=14270 running=3 done=2095 failed=0 blocked=0 pending=12172 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s999:3116s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s999:644s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s888:203s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260809-23:55:51 (UTC+8)
822tasks total=14270 running=3 done=2094 failed=0 blocked=0 pending=12173 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s999:2749s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s999:278s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s888:909s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:98%260809-23:49:45 (UTC+8)
821tasks total=14270 running=3 done=2093 failed=0 blocked=0 pending=12174 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s999:2383s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s999:1416s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s888:542s | GPU cuda1:5445/24564MiB:95%;cuda2:7851/24564MiB:0%;cuda3:7851/24564MiB:97%260809-23:43:38 (UTC+8)
820tasks total=14270 running=3 done=2093 failed=0 blocked=0 pending=12174 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s999:2200s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s999:1233s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s888:359s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260809-23:40:35 (UTC+8)
819tasks total=14270 running=3 done=2091 failed=0 blocked=0 pending=12176 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s999:1830s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s999:863s;cuda3:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s888:1445s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260809-23:34:28 (UTC+8)
818tasks total=14270 running=3 done=2088 failed=0 blocked=0 pending=12179 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s999:903s;cuda2:trace_context:trace_bank__base__Semantic-128-b__llama3_8b__wikics__s999:851s;cuda3:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s888:517s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:97%260809-23:18:58 (UTC+8)
817tasks total=14270 running=3 done=2087 failed=0 blocked=0 pending=12180 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s999:349s;cuda2:trace_context:trace_bank__base__Semantic-128-b__llama3_8b__wikics__s999:297s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s888:1416s | GPU cuda1:5445/24564MiB:0%;cuda2:7851/24564MiB:0%;cuda3:7851/24564MiB:97%260809-23:09:44 (UTC+8)
816tasks total=14270 running=3 done=2083 failed=0 blocked=0 pending=12184 | results done=707/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s999:2890s;cuda2:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s999:657s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s888:864s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:97%260809-23:00:32 (UTC+8)
815tasks total=14270 running=3 done=2076 failed=0 blocked=0 pending=12191 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s999:1770s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s999:1901s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s888:1998s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:96%260809-22:41:52 (UTC+8)
814tasks total=14270 running=3 done=2076 failed=0 blocked=0 pending=12191 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s999:1588s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s999:1719s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s888:1817s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:96%260809-22:38:51 (UTC+8)
813tasks total=14270 running=3 done=2076 failed=0 blocked=0 pending=12191 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s999:1407s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s999:1538s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s888:1636s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:96%260809-22:35:49 (UTC+8)
812tasks total=14270 running=3 done=2076 failed=0 blocked=0 pending=12191 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s999:854s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s999:985s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s888:1083s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:96%260809-22:26:37 (UTC+8)
811tasks total=14270 running=3 done=2073 failed=0 blocked=0 pending=12194 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s999:2762s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s999:2224s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s888:2119s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:99%260809-22:07:53 (UTC+8)
810tasks total=14270 running=3 done=2073 failed=0 blocked=0 pending=12194 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s999:2020s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s999:1482s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s888:1377s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:27%;cuda3:7851/24564MiB:96%260809-21:55:34 (UTC+8)
809tasks total=14270 running=3 done=2073 failed=0 blocked=0 pending=12194 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s999:1466s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s999:928s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s888:823s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:96%260809-21:46:17 (UTC+8)
808tasks total=14270 running=3 done=2073 failed=0 blocked=0 pending=12194 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s999:914s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s999:376s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s888:271s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:96%260809-21:37:05 (UTC+8)
807tasks total=14270 running=3 done=2071 failed=0 blocked=0 pending=12196 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s999:547s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s999:9s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s888:1254s | GPU cuda1:4885/24564MiB:0%;cuda2:7645/24564MiB:13%;cuda3:7851/24564MiB:97%260809-21:30:59 (UTC+8)
806tasks total=14270 running=3 done=2068 failed=0 blocked=0 pending=12199 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s999:366s;cuda2:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s999:1310s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s888:1073s | GPU cuda1:4885/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260809-21:27:57 (UTC+8)
805tasks total=14270 running=3 done=2065 failed=0 blocked=0 pending=12202 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s999:2576s;cuda2:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s999:753s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s888:516s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:98%260809-21:18:41 (UTC+8)
804tasks total=14270 running=3 done=2064 failed=0 blocked=0 pending=12203 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s999:2020s;cuda2:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s999:196s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s888:1305s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260809-21:09:24 (UTC+8)
803tasks total=14270 running=3 done=2064 failed=0 blocked=0 pending=12203 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s999:1839s;cuda2:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s999:15s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s888:1124s | GPU cuda1:4625/24564MiB:0%;cuda2:7689/24564MiB:13%;cuda3:7851/24564MiB:97%260809-21:06:23 (UTC+8)
802tasks total=14270 running=3 done=2063 failed=0 blocked=0 pending=12204 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s999:1472s;cuda2:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s999:1126s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s888:758s | GPU cuda1:4625/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260809-21:00:16 (UTC+8)
801tasks total=14270 running=3 done=2063 failed=0 blocked=0 pending=12204 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s999:1103s;cuda2:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s999:757s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s888:389s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:0%260809-20:54:08 (UTC+8)
800tasks total=14270 running=3 done=2063 failed=0 blocked=0 pending=12204 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s999:922s;cuda2:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s999:576s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s888:208s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260809-20:51:07 (UTC+8)
799tasks total=14270 running=3 done=2063 failed=0 blocked=0 pending=12204 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s999:741s;cuda2:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s999:395s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s888:27s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:97%260809-20:48:06 (UTC+8)
798tasks total=14270 running=3 done=2061 failed=0 blocked=0 pending=12206 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s999:560s;cuda2:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s999:214s;cuda3:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s888:772s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:96%260809-20:45:05 (UTC+8)
797tasks total=14270 running=3 done=2058 failed=0 blocked=0 pending=12209 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s999:4s;cuda2:trace_context:trace_bank__base__Random-256-b__llama3_8b__wikics__s999:728s;cuda3:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s888:215s | GPU cuda1:401/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:91%260809-20:35:48 (UTC+8)
796tasks total=14270 running=3 done=2055 failed=0 blocked=0 pending=12212 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s999:1800s;cuda2:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s999:866s;cuda3:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s888:211s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:95%260809-20:20:20 (UTC+8)
795tasks total=14270 running=3 done=2053 failed=0 blocked=0 pending=12214 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s999:1058s;cuda2:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s999:124s;cuda3:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s888:264s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:98%260809-20:07:58 (UTC+8)
794tasks total=14270 running=3 done=2049 failed=0 blocked=0 pending=12218 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s999:507s;cuda2:trace_context:trace_bank__base__Random-128-b__llama3_8b__wikics__s999:460s;cuda3:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s888:506s | GPU cuda1:4625/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:98%260809-19:58:47 (UTC+8)
793tasks total=14270 running=3 done=2043 failed=0 blocked=0 pending=12224 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s999:2550s;cuda2:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s999:792s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s888:4698s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:96%260809-19:49:35 (UTC+8)
792tasks total=14270 running=3 done=2043 failed=0 blocked=0 pending=12224 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s999:2369s;cuda2:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s999:611s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s888:4517s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:98%;cuda3:7851/24564MiB:0%260809-19:46:34 (UTC+8)
791tasks total=14270 running=3 done=2043 failed=0 blocked=0 pending=12224 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s999:2188s;cuda2:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s999:430s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s888:4335s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:100%260809-19:43:33 (UTC+8)
790tasks total=14270 running=3 done=2040 failed=0 blocked=0 pending=12227 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s999:890s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s999:1490s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s888:3037s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:0%260809-19:21:55 (UTC+8)
789tasks total=14270 running=3 done=2040 failed=0 blocked=0 pending=12227 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s999:338s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s999:938s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s888:2486s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:96%260809-19:12:43 (UTC+8)
788tasks total=14270 running=3 done=2039 failed=0 blocked=0 pending=12228 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s999:2217s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s999:196s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s888:1744s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:0%260809-19:00:21 (UTC+8)
787tasks total=14270 running=3 done=2038 failed=0 blocked=0 pending=12229 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s999:1660s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s999:1994s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s888:1187s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:0%260809-18:51:04 (UTC+8)
786tasks total=14270 running=3 done=2038 failed=0 blocked=0 pending=12229 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s999:537s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s999:871s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s888:64s | GPU cuda1:4565/24564MiB:0%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:96%260809-18:32:21 (UTC+8)
785tasks total=14270 running=3 done=2034 failed=0 blocked=0 pending=12233 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s999:1552s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s999:314s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:4251s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:95%260809-18:23:05 (UTC+8)
784tasks total=14270 running=3 done=2031 failed=0 blocked=0 pending=12236 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s999:1180s;cuda2:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s999:1009s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:3880s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260809-18:16:53 (UTC+8)
783tasks total=14270 running=3 done=2031 failed=0 blocked=0 pending=12236 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s999:629s;cuda2:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s999:457s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:3328s | GPU cuda1:5445/24564MiB:76%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:0%260809-18:07:42 (UTC+8)
782tasks total=14270 running=3 done=2031 failed=0 blocked=0 pending=12236 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s999:260s;cuda2:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s999:88s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:2959s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:95%260809-18:01:33 (UTC+8)
781tasks total=14270 running=3 done=2029 failed=0 blocked=0 pending=12238 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s999:899s;cuda2:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s999:226s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:2032s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260809-17:46:06 (UTC+8)
780tasks total=14270 running=3 done=2026 failed=0 blocked=0 pending=12241 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s999:346s;cuda2:trace_context:trace_bank__base__Full-128-b__llama3_8b__wikics__s999:563s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:1480s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:0%260809-17:36:54 (UTC+8)
779tasks total=14270 running=3 done=2023 failed=0 blocked=0 pending=12244 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s999:721s;cuda2:trace_context:trace_bank__base__Full-128-b__llama3_8b__wikics__s999:11s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:928s | GPU cuda1:4885/24564MiB:95%;cuda2:7655/24564MiB:0%;cuda3:7851/24564MiB:0%260809-17:27:42 (UTC+8)
778tasks total=14270 running=3 done=2022 failed=0 blocked=0 pending=12245 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s999:540s;cuda2:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s999:715s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:747s | GPU cuda1:4885/24564MiB:100%;cuda2:7851/24564MiB:98%;cuda3:7851/24564MiB:100%260809-17:24:41 (UTC+8)
777tasks total=14270 running=3 done=2022 failed=0 blocked=0 pending=12245 | results done=706/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s999:359s;cuda2:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s999:534s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:566s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:0%260809-17:21:40 (UTC+8)
776tasks total=14270 running=3 done=2019 failed=0 blocked=0 pending=12248 | results done=705/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s999:730s;cuda2:fixed_downstream:run__T2_FixedSemantic256__llama3_8b__wikics__s999__sage3:4s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s888:14s | GPU cuda1:4885/24564MiB:96%;cuda2:1423/24564MiB:0%;cuda3:7681/24564MiB:13%260809-17:12:28 (UTC+8)
775tasks total=14270 running=3 done=2010 failed=0 blocked=0 pending=12257 | results done=700/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s999:364s;cuda2:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s999:831s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s888:3505s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:0%260809-17:06:22 (UTC+8)
774tasks total=14270 running=3 done=2007 failed=0 blocked=0 pending=12260 | results done=700/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__wikics__s999:524s;cuda2:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s999:463s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s888:3137s | GPU cuda1:14/24564MiB:0%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:0%260809-17:00:13 (UTC+8)
773tasks total=14270 running=3 done=2007 failed=0 blocked=0 pending=12260 | results done=700/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__wikics__s999:157s;cuda2:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s999:95s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s888:2769s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:100%260809-16:54:07 (UTC+8)
772tasks total=14270 running=3 done=2005 failed=0 blocked=0 pending=12262 | results done=700/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__wikics__s999:127s;cuda2:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s999:633s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s888:2215s | GPU cuda1:4565/24564MiB:97%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:0%260809-16:44:52 (UTC+8)
771tasks total=14270 running=3 done=2002 failed=0 blocked=0 pending=12265 | results done=700/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s999:1303s;cuda2:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s999:265s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s888:1846s | GPU cuda1:5445/24564MiB:94%;cuda2:7851/24564MiB:95%;cuda3:7851/24564MiB:0%260809-16:38:43 (UTC+8)
770tasks total=14270 running=3 done=2002 failed=0 blocked=0 pending=12265 | results done=700/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s999:1122s;cuda2:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s999:84s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s888:1665s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:0%;cuda3:7851/24564MiB:0%260809-16:35:42 (UTC+8)
769tasks total=14270 running=3 done=1993 failed=0 blocked=0 pending=12274 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s999:569s;cuda2:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s999:1134s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s888:1112s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:97%260809-16:26:30 (UTC+8)
768tasks total=14270 running=3 done=1993 failed=0 blocked=0 pending=12274 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s999:14s;cuda2:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s999:579s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s888:558s | GPU cuda1:4457/24564MiB:22%;cuda2:7851/24564MiB:99%;cuda3:7851/24564MiB:100%260809-16:17:17 (UTC+8)
767tasks total=14270 running=3 done=1992 failed=0 blocked=0 pending=12275 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s999:1002s;cuda2:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s999:28s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s888:6s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:401/24564MiB:0%260809-16:08:02 (UTC+8)
766tasks total=14270 running=3 done=1990 failed=0 blocked=0 pending=12277 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s999:450s;cuda2:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s999:954s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s888:3331s | GPU cuda1:5445/24564MiB:100%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:0%260809-15:58:51 (UTC+8)
765tasks total=14270 running=3 done=1987 failed=0 blocked=0 pending=12280 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s999:608s;cuda2:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s999:211s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s888:2588s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:97%;cuda3:7851/24564MiB:0%260809-15:46:28 (UTC+8)
764tasks total=14270 running=3 done=1986 failed=0 blocked=0 pending=12281 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s999:57s;cuda2:trace_context:trace_bank__base__Self__llama3_8b__wikics__s999:391s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s888:2036s | GPU cuda1:4885/24564MiB:96%;cuda2:7851/24564MiB:98%;cuda3:7851/24564MiB:97%260809-15:37:16 (UTC+8)
763tasks total=14270 running=3 done=1984 failed=0 blocked=0 pending=12283 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s999:211s;cuda2:sft:sft__trace__llama3_8b__wikics__s999:3633s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s888:1294s | GPU cuda1:4885/24564MiB:100%;cuda2:10477/24564MiB:99%;cuda3:7851/24564MiB:0%260809-15:24:54 (UTC+8)
762tasks total=14270 running=3 done=1981 failed=0 blocked=0 pending=12286 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-b__llama31_4b__wikics__s999:294s;cuda2:sft:sft__trace__llama3_8b__wikics__s999:3079s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s888:740s | GPU cuda1:4625/24564MiB:97%;cuda2:10477/24564MiB:100%;cuda3:7851/24564MiB:97%260809-15:15:40 (UTC+8)
761tasks total=14270 running=3 done=1980 failed=0 blocked=0 pending=12287 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s999:559s;cuda2:sft:sft__trace__llama3_8b__wikics__s999:2712s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s888:373s | GPU cuda1:4625/24564MiB:97%;cuda2:10477/24564MiB:100%;cuda3:7851/24564MiB:0%260809-15:09:33 (UTC+8)
760tasks total=14270 running=3 done=1980 failed=0 blocked=0 pending=12287 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s999:378s;cuda2:sft:sft__trace__llama3_8b__wikics__s999:2531s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s888:192s | GPU cuda1:4625/24564MiB:97%;cuda2:10477/24564MiB:95%;cuda3:7851/24564MiB:100%260809-15:06:32 (UTC+8)
759tasks total=14270 running=3 done=1980 failed=0 blocked=0 pending=12287 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s999:197s;cuda2:sft:sft__trace__llama3_8b__wikics__s999:2349s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s888:11s | GPU cuda1:4625/24564MiB:97%;cuda2:10477/24564MiB:100%;cuda3:7653/24564MiB:13%260809-15:03:30 (UTC+8)
758tasks total=14270 running=3 done=1978 failed=0 blocked=0 pending=12289 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s999:16s;cuda2:sft:sft__trace__llama3_8b__wikics__s999:2168s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s888:3255s | GPU cuda1:4549/24564MiB:0%;cuda2:10477/24564MiB:99%;cuda3:7851/24564MiB:97%260809-15:00:29 (UTC+8)
757tasks total=14270 running=3 done=1974 failed=0 blocked=0 pending=12293 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-a__llama31_4b__wikics__s999:466s;cuda2:sft:sft__trace__llama3_8b__wikics__s999:1617s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s888:2704s | GPU cuda1:4565/24564MiB:97%;cuda2:10477/24564MiB:95%;cuda3:7851/24564MiB:0%260809-14:51:18 (UTC+8)
756tasks total=14270 running=3 done=1971 failed=0 blocked=0 pending=12296 | results done=694/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s999:1453s;cuda2:sft:sft__trace__llama3_8b__wikics__s999:1060s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s888:2147s | GPU cuda1:5445/24564MiB:93%;cuda2:10477/24564MiB:100%;cuda3:7851/24564MiB:0%260809-14:42:01 (UTC+8)
755tasks total=14270 running=3 done=1968 failed=0 blocked=0 pending=12299 | results done=692/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s999:333s;cuda2:external_downstream:run__LinguGKD__llama31_4b__wikics__s999__sage2:9s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s888:1027s | GPU cuda1:5445/24564MiB:100%;cuda2:1783/24564MiB:20%;cuda3:7851/24564MiB:97%260809-14:23:21 (UTC+8)
754tasks total=14270 running=3 done=1961 failed=0 blocked=0 pending=12306 | results done=688/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s999:1128s;cuda2:lingu_bank:lingu_bank__llama31_4b__wikics__s999:992s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s888:282s | GPU cuda1:5445/24564MiB:94%;cuda2:5545/24564MiB:100%;cuda3:7851/24564MiB:97%260809-14:10:59 (UTC+8)
753tasks total=14270 running=3 done=1960 failed=0 blocked=0 pending=12307 | results done=688/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s999:198s;cuda2:lingu_bank:lingu_bank__llama31_4b__wikics__s999:62s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s888:2814s | GPU cuda1:5445/24564MiB:100%;cuda2:5545/24564MiB:95%;cuda3:7851/24564MiB:0%260809-13:55:26 (UTC+8)
752tasks total=14270 running=3 done=1956 failed=0 blocked=0 pending=12311 | results done=688/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-b__llama31_4b__wikics__s999:91s;cuda2:sft:sft__lingu__llama31_4b__wikics__s999:4168s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s888:2072s | GPU cuda1:4625/24564MiB:97%;cuda2:9171/24564MiB:98%;cuda3:7851/24564MiB:0%260809-13:43:04 (UTC+8)
751tasks total=14270 running=3 done=1955 failed=0 blocked=0 pending=12312 | results done=688/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama31_4b__wikics__s999:357s;cuda2:sft:sft__lingu__llama31_4b__wikics__s999:3801s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s888:1706s | GPU cuda1:4625/24564MiB:97%;cuda2:9171/24564MiB:100%;cuda3:7851/24564MiB:0%260809-13:36:58 (UTC+8)
750tasks total=14270 running=3 done=1951 failed=0 blocked=0 pending=12316 | results done=688/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-a__llama31_4b__wikics__s999:427s;cuda2:sft:sft__lingu__llama31_4b__wikics__s999:2872s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s888:776s | GPU cuda1:4565/24564MiB:94%;cuda2:9169/24564MiB:100%;cuda3:7851/24564MiB:97%260809-13:21:28 (UTC+8)
749tasks total=14270 running=3 done=1942 failed=0 blocked=0 pending=12325 | results done=682/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__wikics__s999:642s;cuda2:sft:sft__lingu__llama31_4b__wikics__s999:2320s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s888:225s | GPU cuda1:4625/24564MiB:97%;cuda2:9169/24564MiB:99%;cuda3:7851/24564MiB:0%260809-13:12:17 (UTC+8)
748tasks total=14270 running=3 done=1940 failed=0 blocked=0 pending=12327 | results done=682/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama31_4b__wikics__s999:275s;cuda2:sft:sft__lingu__llama31_4b__wikics__s999:1954s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s888:3151s | GPU cuda1:4625/24564MiB:97%;cuda2:9169/24564MiB:100%;cuda3:7851/24564MiB:0%260809-13:06:10 (UTC+8)
747tasks total=14270 running=3 done=1939 failed=0 blocked=0 pending=12328 | results done=682/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__wikics__s999:189s;cuda2:sft:sft__lingu__llama31_4b__wikics__s999:1209s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s888:2407s | GPU cuda1:4625/24564MiB:97%;cuda2:9169/24564MiB:58%;cuda3:7851/24564MiB:95%260809-12:53:46 (UTC+8)
746tasks total=14270 running=3 done=1924 failed=0 blocked=0 pending=12343 | results done=675/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s999:602s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s888:1303s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s888:909s | GPU cuda1:4885/24564MiB:96%;cuda2:5485/24564MiB:51%;cuda3:7851/24564MiB:0%260809-12:28:48 (UTC+8)
745tasks total=14270 running=3 done=1924 failed=0 blocked=0 pending=12343 | results done=675/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s999:50s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s888:751s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s888:357s | GPU cuda1:4885/24564MiB:100%;cuda2:5485/24564MiB:98%;cuda3:7851/24564MiB:0%260809-12:19:36 (UTC+8)
744tasks total=14270 running=3 done=1921 failed=0 blocked=0 pending=12346 | results done=675/5050 | RUNNING cuda1:sft:sft__trace__llama31_4b__wikics__s999:2544s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s888:199s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s888:3040s | GPU cuda1:7969/24564MiB:100%;cuda2:5445/24564MiB:98%;cuda3:7851/24564MiB:98%260809-12:10:25 (UTC+8)
743tasks total=14270 running=3 done=1920 failed=0 blocked=0 pending=12347 | results done=675/5050 | RUNNING cuda1:sft:sft__trace__llama31_4b__wikics__s999:1992s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s888:1135s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s888:2488s | GPU cuda1:7967/24564MiB:100%;cuda2:5485/24564MiB:95%;cuda3:7851/24564MiB:0%260809-12:01:13 (UTC+8)
742tasks total=14270 running=3 done=1920 failed=0 blocked=0 pending=12347 | results done=675/5050 | RUNNING cuda1:sft:sft__trace__llama31_4b__wikics__s999:1625s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s888:769s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s888:2122s | GPU cuda1:7967/24564MiB:99%;cuda2:5485/24564MiB:100%;cuda3:7851/24564MiB:97%260809-11:55:06 (UTC+8)
741tasks total=14270 running=3 done=1920 failed=0 blocked=0 pending=12347 | results done=675/5050 | RUNNING cuda1:sft:sft__trace__llama31_4b__wikics__s999:1074s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s888:217s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s888:1570s | GPU cuda1:7967/24564MiB:100%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:0%260809-11:45:55 (UTC+8)
740tasks total=14270 running=3 done=1918 failed=0 blocked=0 pending=12349 | results done=675/5050 | RUNNING cuda1:sft:sft__trace__llama31_4b__wikics__s999:139s;cuda2:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s888:179s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s888:636s | GPU cuda1:7967/24564MiB:99%;cuda2:4885/24564MiB:97%;cuda3:7851/24564MiB:0%260809-11:30:20 (UTC+8)
739tasks total=14270 running=3 done=1906 failed=0 blocked=0 pending=12361 | results done=669/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:1718s;cuda2:trace_context:trace_bank__n2__HopQuota-512-a__llama31_4b__wikics__s888:148s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s888:2055s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:100%260809-11:14:53 (UTC+8)
738tasks total=14270 running=3 done=1904 failed=0 blocked=0 pending=12363 | results done=669/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:976s;cuda2:trace_context:trace_bank__n2__HopQuota-256-b__llama31_4b__wikics__s888:22s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s888:1313s | GPU cuda1:7851/24564MiB:95%;cuda2:4585/24564MiB:97%;cuda3:7851/24564MiB:100%260809-11:02:31 (UTC+8)
737tasks total=14270 running=3 done=1903 failed=0 blocked=0 pending=12364 | results done=669/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:795s;cuda2:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s888:454s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s888:1132s | GPU cuda1:7851/24564MiB:96%;cuda2:4605/24564MiB:97%;cuda3:7851/24564MiB:95%260809-10:59:30 (UTC+8)
736tasks total=14270 running=3 done=1903 failed=0 blocked=0 pending=12364 | results done=669/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:614s;cuda2:trace_context:trace_bank__n2__HopQuota-256-a__llama31_4b__wikics__s888:273s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s888:951s | GPU cuda1:7851/24564MiB:100%;cuda2:4605/24564MiB:97%;cuda3:7851/24564MiB:96%260809-10:56:29 (UTC+8)
735tasks total=14270 running=3 done=1901 failed=0 blocked=0 pending=12366 | results done=669/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:245s;cuda2:trace_context:trace_bank__n2__HopQuota-128-b__llama31_4b__wikics__s888:372s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s888:582s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:94%;cuda3:7851/24564MiB:95%260809-10:50:20 (UTC+8)
734tasks total=14270 running=3 done=1901 failed=0 blocked=0 pending=12366 | results done=669/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama3_8b__wikics__s888:64s;cuda2:trace_context:trace_bank__n2__HopQuota-128-b__llama31_4b__wikics__s888:191s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s888:401s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:94%;cuda3:7851/24564MiB:96%260809-10:47:19 (UTC+8)
733tasks total=14270 running=3 done=1896 failed=0 blocked=0 pending=12371 | results done=669/5050 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:5493s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s888:3270s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s888:1638s | GPU cuda1:11839/24564MiB:100%;cuda2:5465/24564MiB:0%;cuda3:7851/24564MiB:96%260809-10:28:51 (UTC+8)
732tasks total=14270 running=3 done=1896 failed=0 blocked=0 pending=12371 | results done=669/5050 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:4941s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s888:2718s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s888:1086s | GPU cuda1:11839/24564MiB:99%;cuda2:5465/24564MiB:0%;cuda3:7851/24564MiB:96%260809-10:19:40 (UTC+8)
731tasks total=14270 running=3 done=1896 failed=0 blocked=0 pending=12371 | results done=669/5050 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:4390s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s888:2166s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s888:534s | GPU cuda1:11839/24564MiB:100%;cuda2:5465/24564MiB:0%;cuda3:7851/24564MiB:100%260809-10:10:28 (UTC+8)
730tasks total=14270 running=3 done=1894 failed=0 blocked=0 pending=12373 | results done=669/5050 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:3648s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s888:1424s;cuda3:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s888:1273s | GPU cuda1:11839/24564MiB:100%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:97%260809-09:58:06 (UTC+8)
729tasks total=14270 running=3 done=1890 failed=0 blocked=0 pending=12377 | results done=669/5050 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:850s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s888:2300s;cuda3:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s888:852s | GPU cuda1:11839/24564MiB:100%;cuda2:5465/24564MiB:0%;cuda3:7851/24564MiB:95%260809-09:11:28 (UTC+8)
728tasks total=14270 running=3 done=1890 failed=0 blocked=0 pending=12377 | results done=669/5050 | RUNNING cuda1:sft:sft__lingu__llama3_8b__wikics__s888:108s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s888:1558s;cuda3:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s888:110s | GPU cuda1:401/24564MiB:0%;cuda2:5465/24564MiB:95%;cuda3:7851/24564MiB:98%260809-08:59:06 (UTC+8)
727tasks total=14270 running=3 done=1884 failed=0 blocked=0 pending=12383 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s666:1866s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s888:1006s;cuda3:trace_context:trace_bank__n1__Semantic-128-a__llama3_8b__wikics__s888:457s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260809-08:49:55 (UTC+8)
726tasks total=14270 running=3 done=1884 failed=0 blocked=0 pending=12383 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s666:1499s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s888:640s;cuda3:trace_context:trace_bank__n1__Semantic-128-a__llama3_8b__wikics__s888:91s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260809-08:43:48 (UTC+8)
725tasks total=14270 running=3 done=1882 failed=0 blocked=0 pending=12385 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s666:1316s;cuda2:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s888:456s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s888:2226s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:95%260809-08:40:47 (UTC+8)
724tasks total=14270 running=3 done=1880 failed=0 blocked=0 pending=12387 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s666:18s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s888:2226s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s888:928s | GPU cuda1:7703/24564MiB:23%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:100%260809-08:19:07 (UTC+8)
723tasks total=14270 running=3 done=1879 failed=0 blocked=0 pending=12388 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s666:1670s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s888:1672s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s888:374s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:100%260809-08:09:53 (UTC+8)
722tasks total=14270 running=3 done=1879 failed=0 blocked=0 pending=12388 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s666:1303s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s888:1306s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s888:8s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7633/24564MiB:13%260809-08:03:46 (UTC+8)
721tasks total=14270 running=3 done=1878 failed=0 blocked=0 pending=12389 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s666:1122s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s888:1125s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s888:2144s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260809-08:00:45 (UTC+8)
720tasks total=14270 running=3 done=1878 failed=0 blocked=0 pending=12389 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s666:751s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s888:753s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s888:1773s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260809-07:54:34 (UTC+8)
719tasks total=14270 running=3 done=1878 failed=0 blocked=0 pending=12389 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s666:570s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s888:573s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s888:1592s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260809-07:51:33 (UTC+8)
718tasks total=14270 running=3 done=1878 failed=0 blocked=0 pending=12389 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s666:389s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s888:392s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s888:1411s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260809-07:48:32 (UTC+8)
717tasks total=14270 running=3 done=1878 failed=0 blocked=0 pending=12389 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s666:208s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s888:211s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s888:1230s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:100%;cuda3:7851/24564MiB:100%260809-07:45:31 (UTC+8)
716tasks total=14270 running=3 done=1875 failed=0 blocked=0 pending=12392 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s666:1213s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s888:2859s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s888:859s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:95%260809-07:39:20 (UTC+8)
715tasks total=14270 running=3 done=1873 failed=0 blocked=0 pending=12394 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s666:288s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s888:1935s;cuda3:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s888:1388s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:99%;cuda3:7851/24564MiB:100%260809-07:23:55 (UTC+8)
714tasks total=14270 running=3 done=1873 failed=0 blocked=0 pending=12394 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s666:107s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s888:1754s;cuda3:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s888:1207s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:100%260809-07:20:54 (UTC+8)
713tasks total=14270 running=3 done=1872 failed=0 blocked=0 pending=12395 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s666:1300s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s888:1573s;cuda3:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s888:1026s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260809-07:17:54 (UTC+8)
712tasks total=14270 running=3 done=1872 failed=0 blocked=0 pending=12395 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s666:548s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s888:821s;cuda3:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s888:274s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260809-07:05:22 (UTC+8)
711tasks total=14270 running=3 done=1871 failed=0 blocked=0 pending=12396 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s666:177s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s888:449s;cuda3:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s888:1355s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260809-06:59:10 (UTC+8)
710tasks total=14270 running=3 done=1867 failed=0 blocked=0 pending=12400 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s666:391s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s888:2452s;cuda3:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s888:613s | GPU cuda1:7851/24564MiB:95%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:97%260809-06:46:48 (UTC+8)
709tasks total=14270 running=3 done=1867 failed=0 blocked=0 pending=12400 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s666:25s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s888:2086s;cuda3:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s888:246s | GPU cuda1:7851/24564MiB:66%;cuda2:4625/24564MiB:99%;cuda3:7851/24564MiB:96%260809-06:40:42 (UTC+8)
708tasks total=14270 running=3 done=1864 failed=0 blocked=0 pending=12403 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-256-a__llama3_8b__wikics__s666:243s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s888:1349s;cuda3:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s888:562s | GPU cuda1:7851/24564MiB:95%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:94%260809-06:28:25 (UTC+8)
707tasks total=14270 running=3 done=1862 failed=0 blocked=0 pending=12405 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-128-b__llama3_8b__wikics__s666:692s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s888:982s;cuda3:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s888:196s | GPU cuda1:7851/24564MiB:94%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:95%260809-06:22:19 (UTC+8)
706tasks total=14270 running=3 done=1860 failed=0 blocked=0 pending=12407 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-128-a__llama3_8b__wikics__s666:766s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s888:243s;cuda3:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s888:507s | GPU cuda1:7851/24564MiB:95%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:97%260809-06:10:00 (UTC+8)
705tasks total=14270 running=3 done=1855 failed=0 blocked=0 pending=12412 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s666:3863s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s888:1690s;cuda3:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s888:70s | GPU cuda1:7851/24564MiB:0%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:98%260809-05:48:09 (UTC+8)
704tasks total=14270 running=3 done=1852 failed=0 blocked=0 pending=12415 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s666:2755s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s888:582s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s888:2152s | GPU cuda1:7851/24564MiB:0%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:96%260809-05:29:42 (UTC+8)
703tasks total=14270 running=3 done=1852 failed=0 blocked=0 pending=12415 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s666:2204s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s888:31s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s888:1600s | GPU cuda1:7851/24564MiB:100%;cuda2:4549/24564MiB:0%;cuda3:7851/24564MiB:100%260809-05:20:30 (UTC+8)
702tasks total=14270 running=3 done=1850 failed=0 blocked=0 pending=12417 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s666:1652s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s888:2100s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s888:1049s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260809-05:11:19 (UTC+8)
701tasks total=14270 running=3 done=1850 failed=0 blocked=0 pending=12417 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s666:1100s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s888:1549s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s888:497s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260809-05:02:07 (UTC+8)
700tasks total=14270 running=3 done=1849 failed=0 blocked=0 pending=12418 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s666:549s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s888:997s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s888:2260s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:96%260809-04:52:55 (UTC+8)
699tasks total=14270 running=3 done=1848 failed=0 blocked=0 pending=12419 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:4407s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s888:435s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s888:1699s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260809-04:43:34 (UTC+8)
698tasks total=14270 running=3 done=1848 failed=0 blocked=0 pending=12419 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:4226s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s888:254s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s888:1518s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260809-04:40:33 (UTC+8)
697tasks total=14270 running=3 done=1847 failed=0 blocked=0 pending=12420 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:3672s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s888:2290s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s888:964s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:95%260809-04:31:19 (UTC+8)
696tasks total=14270 running=3 done=1847 failed=0 blocked=0 pending=12420 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:2930s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s888:1549s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s888:222s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:94%;cuda3:7851/24564MiB:96%260809-04:18:57 (UTC+8)
695tasks total=14270 running=3 done=1845 failed=0 blocked=0 pending=12422 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:2559s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s888:1177s;cuda3:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s888:901s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:97%260809-04:12:46 (UTC+8)
694tasks total=14270 running=3 done=1845 failed=0 blocked=0 pending=12422 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:2378s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s888:996s;cuda3:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s888:720s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:97%260809-04:09:45 (UTC+8)
693tasks total=14270 running=3 done=1844 failed=0 blocked=0 pending=12423 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:1639s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s888:257s;cuda3:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s888:1028s | GPU cuda1:7851/24564MiB:99%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:97%260809-03:57:25 (UTC+8)
692tasks total=14270 running=3 done=1842 failed=0 blocked=0 pending=12425 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:897s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s888:1109s;cuda3:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s888:286s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260809-03:45:04 (UTC+8)
691tasks total=14270 running=3 done=1842 failed=0 blocked=0 pending=12425 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:716s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s888:928s;cuda3:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s888:105s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:94%;cuda3:7851/24564MiB:97%260809-03:42:03 (UTC+8)
690tasks total=14270 running=3 done=1840 failed=0 blocked=0 pending=12427 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:535s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s888:747s;cuda3:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s888:797s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:95%260809-03:39:02 (UTC+8)
689tasks total=14270 running=3 done=1840 failed=0 blocked=0 pending=12427 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:354s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s888:566s;cuda3:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s888:616s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:98%260809-03:36:01 (UTC+8)
688tasks total=14270 running=3 done=1840 failed=0 blocked=0 pending=12427 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s666:173s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s888:385s;cuda3:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s888:435s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:98%260809-03:33:00 (UTC+8)
687tasks total=14270 running=3 done=1836 failed=0 blocked=0 pending=12431 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s666:3009s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s888:1238s;cuda3:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s888:567s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:98%260809-03:20:41 (UTC+8)
686tasks total=14270 running=3 done=1836 failed=0 blocked=0 pending=12431 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s666:2828s;cuda2:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s888:1057s;cuda3:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s888:386s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:95%260809-03:17:40 (UTC+8)
685tasks total=14270 running=3 done=1828 failed=0 blocked=0 pending=12439 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s666:15s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s888:121s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s888:1182s | GPU cuda1:7687/24564MiB:22%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:96%260809-02:30:47 (UTC+8)
684tasks total=14270 running=3 done=1825 failed=0 blocked=0 pending=12442 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s666:3247s;cuda2:trace_context:trace_bank__n2__Semantic-128-b__llama31_4b__wikics__s888:289s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s888:816s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:97%260809-02:24:41 (UTC+8)
683tasks total=14270 running=3 done=1824 failed=0 blocked=0 pending=12443 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s666:2693s;cuda2:trace_context:trace_bank__n2__Semantic-128-a__llama31_4b__wikics__s888:268s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s888:262s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:94%;cuda3:7851/24564MiB:97%260809-02:15:27 (UTC+8)
682tasks total=14270 running=3 done=1824 failed=0 blocked=0 pending=12443 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s666:2512s;cuda2:trace_context:trace_bank__n2__Semantic-128-a__llama31_4b__wikics__s888:87s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s888:81s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:100%260809-02:12:26 (UTC+8)
681tasks total=14270 running=3 done=1821 failed=0 blocked=0 pending=12446 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s666:2146s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s888:1291s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s888:1166s | GPU cuda1:7851/24564MiB:0%;cuda2:5465/24564MiB:94%;cuda3:7851/24564MiB:100%260809-02:06:19 (UTC+8)
680tasks total=14270 running=3 done=1821 failed=0 blocked=0 pending=12446 | results done=668/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s666:1594s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s888:739s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s888:614s | GPU cuda1:7851/24564MiB:0%;cuda2:5465/24564MiB:94%;cuda3:7851/24564MiB:96%260809-01:57:08 (UTC+8)
679tasks total=14270 running=3 done=1818 failed=0 blocked=0 pending=12449 | results done=667/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s666:846s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s888:1557s;cuda3:g4_teacher:g4_teacher__llama3_8b__wikics__s888:613s | GPU cuda1:7851/24564MiB:97%;cuda2:5465/24564MiB:100%;cuda3:7851/24564MiB:96%260809-01:44:40 (UTC+8)
678tasks total=14270 running=3 done=1808 failed=0 blocked=0 pending=12459 | results done=660/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s666:3077s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s888:630s;cuda3:trace_downstream:run__O1_OrderA__llama3_8b__wikics__s888__gcn3:41s | GPU cuda1:7851/24564MiB:0%;cuda2:5465/24564MiB:94%;cuda3:1637/24564MiB:80%260809-01:29:13 (UTC+8)
677tasks total=14270 running=3 done=1791 failed=0 blocked=0 pending=12476 | results done=643/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s666:2526s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s888:78s;cuda3:trace_downstream:run__C2_Semantic__llama3_8b__wikics__s888__gcn3:29s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:95%;cuda3:1637/24564MiB:81%260809-01:20:01 (UTC+8)
676tasks total=14270 running=3 done=1754 failed=0 blocked=0 pending=12513 | results done=608/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s666:1596s;cuda2:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s888:63s;cuda3:trace_downstream:run__T4_StudentOptGlobal__llama3_8b__wikics__s888__sage3:30s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:96%;cuda3:1441/24564MiB:82%260809-01:04:32 (UTC+8)
675tasks total=14270 running=3 done=1746 failed=0 blocked=0 pending=12521 | results done=601/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s666:1044s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s888:421s;cuda3:trace_downstream:run__T3_TeacherOptGlobal__llama3_8b__wikics__s888__sage2:14s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:1435/24564MiB:72%260809-00:55:20 (UTC+8)
674tasks total=14270 running=3 done=1734 failed=0 blocked=0 pending=12533 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s666:2918s;cuda2:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s888:414s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s888:1016s | GPU cuda1:7851/24564MiB:0%;cuda2:4605/24564MiB:37%;cuda3:7851/24564MiB:100%260809-00:33:45 (UTC+8)
673tasks total=14270 running=3 done=1734 failed=0 blocked=0 pending=12533 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s666:2736s;cuda2:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s888:233s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s888:835s | GPU cuda1:7851/24564MiB:98%;cuda2:4605/24564MiB:97%;cuda3:7851/24564MiB:95%260809-00:30:43 (UTC+8)
672tasks total=14270 running=3 done=1734 failed=0 blocked=0 pending=12533 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s666:2555s;cuda2:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s888:52s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s888:654s | GPU cuda1:7851/24564MiB:97%;cuda2:4605/24564MiB:97%;cuda3:7851/24564MiB:99%260809-00:27:42 (UTC+8)
671tasks total=14270 running=3 done=1732 failed=0 blocked=0 pending=12535 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s666:2004s;cuda2:trace_context:trace_bank__n2__Random-128-b__llama31_4b__wikics__s888:9s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s888:102s | GPU cuda1:7851/24564MiB:97%;cuda2:4423/24564MiB:13%;cuda3:7851/24564MiB:100%260809-00:18:30 (UTC+8)
670tasks total=14270 running=3 done=1728 failed=0 blocked=0 pending=12539 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s666:1260s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s888:1340s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s888:1502s | GPU cuda1:7851/24564MiB:0%;cuda2:5465/24564MiB:100%;cuda3:7851/24564MiB:100%260809-00:06:08 (UTC+8)
669tasks total=14270 running=3 done=1728 failed=0 blocked=0 pending=12539 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s666:894s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s888:974s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s888:1136s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:96%260809-00:00:01 (UTC+8)
668tasks total=14270 running=3 done=1725 failed=0 blocked=0 pending=12542 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s666:2774s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s888:1425s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s888:21s | GPU cuda1:7851/24564MiB:95%;cuda2:5465/24564MiB:100%;cuda3:7727/24564MiB:13%260808-23:41:26 (UTC+8)
667tasks total=14270 running=3 done=1722 failed=0 blocked=0 pending=12545 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s666:1662s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s888:312s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s888:244s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:94%;cuda3:7851/24564MiB:97%260808-23:22:53 (UTC+8)
666tasks total=14270 running=3 done=1722 failed=0 blocked=0 pending=12545 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s666:1481s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s888:131s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s888:63s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260808-23:19:51 (UTC+8)
665tasks total=14270 running=3 done=1718 failed=0 blocked=0 pending=12549 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s666:370s;cuda2:trace_context:trace_bank__n2__Full-256-a__llama31_4b__wikics__s888:308s;cuda3:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s888:284s | GPU cuda1:7851/24564MiB:0%;cuda2:4605/24564MiB:97%;cuda3:7851/24564MiB:97%260808-23:01:21 (UTC+8)
664tasks total=14270 running=3 done=1711 failed=0 blocked=0 pending=12556 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s666:2452s;cuda2:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s888:389s;cuda3:trace_context:trace_bank__base__HopQuota-256-b__llama3_8b__wikics__s888:268s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:98%260808-22:45:49 (UTC+8)
663tasks total=14270 running=3 done=1708 failed=0 blocked=0 pending=12559 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s666:1898s;cuda2:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s888:504s;cuda3:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s888:625s | GPU cuda1:7851/24564MiB:95%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:98%260808-22:36:35 (UTC+8)
662tasks total=14270 running=3 done=1704 failed=0 blocked=0 pending=12563 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s666:1161s;cuda2:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s888:434s;cuda3:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s888:676s | GPU cuda1:7851/24564MiB:98%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:96%260808-22:24:18 (UTC+8)
661tasks total=14270 running=3 done=1702 failed=0 blocked=0 pending=12565 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s666:607s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s888:794s;cuda3:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s888:122s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:95%260808-22:15:04 (UTC+8)
660tasks total=14270 running=3 done=1701 failed=0 blocked=0 pending=12566 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s666:238s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s888:425s;cuda3:trace_context:trace_bank__base__HopQuota-128-a__llama3_8b__wikics__s888:536s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:92%260808-22:08:55 (UTC+8)
659tasks total=14270 running=3 done=1701 failed=0 blocked=0 pending=12566 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s666:56s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s888:243s;cuda3:trace_context:trace_bank__base__HopQuota-128-a__llama3_8b__wikics__s888:354s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:95%260808-22:05:54 (UTC+8)
658tasks total=14270 running=3 done=1695 failed=0 blocked=0 pending=12572 | results done=597/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s666:1668s;cuda2:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s888:415s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s888:4040s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:100%;cuda3:7851/24564MiB:100%260808-21:53:36 (UTC+8)
657tasks total=14270 running=3 done=1691 failed=0 blocked=0 pending=12576 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s666:1115s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s888:1425s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s888:3487s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260808-21:44:22 (UTC+8)
656tasks total=14270 running=3 done=1691 failed=0 blocked=0 pending=12576 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s666:747s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s888:1057s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s888:3119s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:95%260808-21:38:14 (UTC+8)
655tasks total=14270 running=3 done=1691 failed=0 blocked=0 pending=12576 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s666:564s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s888:873s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s888:2935s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:100%260808-21:35:11 (UTC+8)
654tasks total=14270 running=3 done=1691 failed=0 blocked=0 pending=12576 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s666:4s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s888:314s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s888:2376s | GPU cuda1:401/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260808-21:25:57 (UTC+8)
653tasks total=14270 running=3 done=1689 failed=0 blocked=0 pending=12578 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s666:1616s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s888:1039s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s888:1638s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:96%260808-21:13:33 (UTC+8)
652tasks total=14270 running=3 done=1689 failed=0 blocked=0 pending=12578 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s666:1064s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s888:487s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s888:1086s | GPU cuda1:7851/24564MiB:95%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:0%260808-21:04:20 (UTC+8)
651tasks total=14270 running=3 done=1687 failed=0 blocked=0 pending=12580 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s666:512s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s888:822s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s888:534s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:96%260808-20:55:08 (UTC+8)
650tasks total=14270 running=3 done=1683 failed=0 blocked=0 pending=12584 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s666:1067s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s888:774s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s888:4084s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260808-20:39:41 (UTC+8)
649tasks total=14270 running=3 done=1683 failed=0 blocked=0 pending=12584 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s666:700s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s888:408s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s888:3718s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:96%260808-20:33:29 (UTC+8)
648tasks total=14270 running=3 done=1683 failed=0 blocked=0 pending=12584 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s666:326s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s888:34s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s888:3344s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260808-20:27:20 (UTC+8)
647tasks total=14270 running=3 done=1680 failed=0 blocked=0 pending=12587 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s666:1258s;cuda2:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s888:69s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s888:2791s | GPU cuda1:7851/24564MiB:97%;cuda2:4605/24564MiB:96%;cuda3:7851/24564MiB:0%260808-20:18:04 (UTC+8)
646tasks total=14270 running=3 done=1679 failed=0 blocked=0 pending=12588 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s666:892s;cuda2:trace_context:trace_bank__n1__HopQuota-256-a__llama31_4b__wikics__s888:288s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s888:2424s | GPU cuda1:7851/24564MiB:97%;cuda2:4605/24564MiB:97%;cuda3:7851/24564MiB:100%260808-20:11:56 (UTC+8)
645tasks total=14270 running=3 done=1674 failed=0 blocked=0 pending=12593 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s666:860s;cuda2:trace_context:trace_bank__n1__HopQuota-128-a__llama31_4b__wikics__s888:286s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s888:1491s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:98%;cuda3:7851/24564MiB:0%260808-19:56:23 (UTC+8)
644tasks total=14270 running=3 done=1671 failed=0 blocked=0 pending=12596 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s666:825s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s888:3315s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s888:557s | GPU cuda1:7851/24564MiB:98%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:100%260808-19:40:50 (UTC+8)
643tasks total=14270 running=3 done=1671 failed=0 blocked=0 pending=12596 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s666:643s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s888:3133s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s888:375s | GPU cuda1:7851/24564MiB:98%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:0%260808-19:37:48 (UTC+8)
642tasks total=14270 running=3 done=1666 failed=0 blocked=0 pending=12601 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s666:932s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s888:1096s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s888:1868s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260808-19:03:50 (UTC+8)
641tasks total=14270 running=3 done=1666 failed=0 blocked=0 pending=12601 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s666:565s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s888:729s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s888:1501s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:97%260808-18:57:43 (UTC+8)
640tasks total=14270 running=3 done=1664 failed=0 blocked=0 pending=12603 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s666:1768s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s888:3600s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s888:382s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:0%260808-18:39:04 (UTC+8)
639tasks total=14270 running=3 done=1664 failed=0 blocked=0 pending=12603 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s666:1401s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s888:3232s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s888:14s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:0%;cuda3:7677/24564MiB:14%260808-18:32:57 (UTC+8)
638tasks total=14270 running=3 done=1663 failed=0 blocked=0 pending=12604 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s666:662s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s888:2494s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s888:2839s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:97%260808-18:20:39 (UTC+8)
637tasks total=14270 running=3 done=1663 failed=0 blocked=0 pending=12604 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s666:480s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s888:2312s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s888:2656s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260808-18:17:37 (UTC+8)
636tasks total=14270 running=3 done=1663 failed=0 blocked=0 pending=12604 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama3_8b__wikics__s666:111s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s888:1943s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s888:2287s | GPU cuda1:7851/24564MiB:95%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:97%260808-18:11:29 (UTC+8)
635tasks total=14270 running=3 done=1661 failed=0 blocked=0 pending=12606 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s666:831s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s888:1201s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s888:1545s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:0%260808-17:59:04 (UTC+8)
634tasks total=14270 running=3 done=1661 failed=0 blocked=0 pending=12606 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s666:640s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s888:1010s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s888:1354s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260808-17:56:03 (UTC+8)
633tasks total=14270 running=3 done=1660 failed=0 blocked=0 pending=12607 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s666:1359s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s888:270s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s888:614s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:94%;cuda3:7851/24564MiB:0%260808-17:43:34 (UTC+8)
632tasks total=14270 running=3 done=1658 failed=0 blocked=0 pending=12609 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s666:993s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s888:3254s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s888:248s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:100%260808-17:37:26 (UTC+8)
631tasks total=14270 running=3 done=1658 failed=0 blocked=0 pending=12609 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s666:812s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s888:3073s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s888:67s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260808-17:34:26 (UTC+8)
630tasks total=14270 running=3 done=1655 failed=0 blocked=0 pending=12612 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s666:631s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s888:2891s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s888:3063s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260808-17:31:25 (UTC+8)
629tasks total=14270 running=3 done=1655 failed=0 blocked=0 pending=12612 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-a__llama3_8b__wikics__s666:79s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s888:2340s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s888:2511s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260808-17:22:13 (UTC+8)
628tasks total=14270 running=3 done=1653 failed=0 blocked=0 pending=12614 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s666:393s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s888:1593s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s888:1765s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260808-17:09:46 (UTC+8)
627tasks total=14270 running=3 done=1652 failed=0 blocked=0 pending=12615 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s666:718s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s888:858s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s888:1029s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:100%;cuda3:7851/24564MiB:0%260808-16:57:31 (UTC+8)
626tasks total=14270 running=3 done=1652 failed=0 blocked=0 pending=12615 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama3_8b__wikics__s666:162s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s888:303s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s888:474s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:0%260808-16:48:16 (UTC+8)
625tasks total=14270 running=3 done=1648 failed=0 blocked=0 pending=12619 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s666:482s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s888:3155s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s888:3141s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:0%260808-16:38:59 (UTC+8)
624tasks total=14270 running=3 done=1648 failed=0 blocked=0 pending=12619 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s666:299s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s888:2972s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s888:2958s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:23%;cuda3:7851/24564MiB:97%260808-16:35:58 (UTC+8)
623tasks total=14270 running=3 done=1648 failed=0 blocked=0 pending=12619 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-b__llama3_8b__wikics__s666:117s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s888:2790s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s888:2775s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:97%260808-16:32:55 (UTC+8)
622tasks total=14270 running=3 done=1647 failed=0 blocked=0 pending=12620 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s666:63s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s888:1862s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s888:1848s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260808-16:17:26 (UTC+8)
621tasks total=14270 running=3 done=1645 failed=0 blocked=0 pending=12622 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s666:1082s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s888:556s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s888:542s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260808-15:55:40 (UTC+8)
620tasks total=14270 running=3 done=1645 failed=0 blocked=0 pending=12622 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s666:901s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s888:375s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s888:361s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:0%260808-15:52:39 (UTC+8)
619tasks total=14270 running=3 done=1639 failed=0 blocked=0 pending=12628 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s666:2109s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s888:2324s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s888:2223s | GPU cuda1:7851/24564MiB:96%;cuda2:4605/24564MiB:0%;cuda3:7851/24564MiB:0%260808-15:34:06 (UTC+8)
618tasks total=14270 running=3 done=1639 failed=0 blocked=0 pending=12628 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s666:1928s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s888:2142s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s888:2041s | GPU cuda1:7851/24564MiB:96%;cuda2:4605/24564MiB:0%;cuda3:7851/24564MiB:0%260808-15:31:04 (UTC+8)
617tasks total=14270 running=3 done=1639 failed=0 blocked=0 pending=12628 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s666:1179s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s888:1393s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s888:1292s | GPU cuda1:7851/24564MiB:96%;cuda2:4605/24564MiB:0%;cuda3:7851/24564MiB:0%260808-15:18:40 (UTC+8)
616tasks total=14270 running=3 done=1639 failed=0 blocked=0 pending=12628 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s666:813s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s888:1027s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s888:926s | GPU cuda1:7851/24564MiB:96%;cuda2:4605/24564MiB:0%;cuda3:7851/24564MiB:0%260808-15:12:29 (UTC+8)
615tasks total=14270 running=3 done=1639 failed=0 blocked=0 pending=12628 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama3_8b__wikics__s666:260s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s888:474s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s888:373s | GPU cuda1:7851/24564MiB:100%;cuda2:4605/24564MiB:0%;cuda3:7851/24564MiB:95%260808-15:03:17 (UTC+8)
614tasks total=14270 running=3 done=1635 failed=0 blocked=0 pending=12632 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s666:770s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s888:2942s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s888:2814s | GPU cuda1:7851/24564MiB:97%;cuda2:4605/24564MiB:0%;cuda3:7851/24564MiB:0%260808-14:54:04 (UTC+8)
613tasks total=14270 running=3 done=1634 failed=0 blocked=0 pending=12633 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-256-a__llama3_8b__wikics__s666:348s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s888:1461s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s888:1333s | GPU cuda1:7851/24564MiB:97%;cuda2:4605/24564MiB:97%;cuda3:7851/24564MiB:0%260808-14:29:24 (UTC+8)
612tasks total=14270 running=3 done=1632 failed=0 blocked=0 pending=12635 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-b__llama3_8b__wikics__s666:858s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s888:1094s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s888:966s | GPU cuda1:7851/24564MiB:95%;cuda2:4605/24564MiB:0%;cuda3:7851/24564MiB:0%260808-14:23:16 (UTC+8)
611tasks total=14270 running=3 done=1626 failed=0 blocked=0 pending=12641 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-a__llama3_8b__wikics__s666:435s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s888:2784s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s888:2011s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:96%260808-14:01:40 (UTC+8)
610tasks total=14270 running=3 done=1624 failed=0 blocked=0 pending=12643 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s666:774s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s888:2035s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s888:1262s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260808-13:49:13 (UTC+8)
609tasks total=14270 running=3 done=1624 failed=0 blocked=0 pending=12643 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s666:593s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s888:1854s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s888:1081s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260808-13:46:10 (UTC+8)
608tasks total=14270 running=3 done=1624 failed=0 blocked=0 pending=12643 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s666:41s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s888:1303s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s888:529s | GPU cuda1:7851/24564MiB:98%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260808-13:36:59 (UTC+8)
607tasks total=14270 running=3 done=1622 failed=0 blocked=0 pending=12645 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s666:574s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s888:751s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s888:2314s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260808-13:27:47 (UTC+8)
606tasks total=14270 running=3 done=1622 failed=0 blocked=0 pending=12645 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s666:20s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s888:197s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s888:1760s | GPU cuda1:7719/24564MiB:22%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:98%260808-13:18:33 (UTC+8)
605tasks total=14270 running=3 done=1619 failed=0 blocked=0 pending=12648 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s666:739s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s888:2375s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s888:1019s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260808-13:06:12 (UTC+8)
604tasks total=14270 running=3 done=1619 failed=0 blocked=0 pending=12648 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama3_8b__wikics__s666:558s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s888:2193s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s888:838s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:96%260808-13:03:11 (UTC+8)
603tasks total=14270 running=3 done=1618 failed=0 blocked=0 pending=12649 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s666:1456s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s888:1634s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s888:279s | GPU cuda1:4079/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:0%260808-12:53:54 (UTC+8)
602tasks total=14270 running=3 done=1615 failed=0 blocked=0 pending=12652 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s666:529s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s888:707s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s888:829s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:97%260808-12:38:24 (UTC+8)
601tasks total=14270 running=3 done=1615 failed=0 blocked=0 pending=12652 | results done=596/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s666:162s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s888:340s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s888:462s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260808-12:32:18 (UTC+8)
600tasks total=14270 running=3 done=1608 failed=0 blocked=0 pending=12659 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s666:1691s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s888:1202s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s888:1192s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:100%260808-12:19:57 (UTC+8)
599tasks total=14270 running=3 done=1608 failed=0 blocked=0 pending=12659 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s666:958s;cuda2:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s888:469s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s888:458s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260808-12:07:40 (UTC+8)
598tasks total=14270 running=3 done=1603 failed=0 blocked=0 pending=12664 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s666:1827s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s888:764s;cuda3:trace_context:trace_bank__base__Semantic-128-b__llama3_8b__wikics__s888:53s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:95%260808-11:45:56 (UTC+8)
597tasks total=14270 running=3 done=1602 failed=0 blocked=0 pending=12665 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s666:1274s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s888:211s;cuda3:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s888:394s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260808-11:36:45 (UTC+8)
596tasks total=14270 running=3 done=1602 failed=0 blocked=0 pending=12665 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s666:1093s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s888:30s;cuda3:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s888:213s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:95%260808-11:33:42 (UTC+8)
595tasks total=14270 running=3 done=1600 failed=0 blocked=0 pending=12667 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s666:910s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__wikics__s888:795s;cuda3:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s888:29s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:98%;cuda3:7851/24564MiB:95%260808-11:30:41 (UTC+8)
594tasks total=14270 running=3 done=1597 failed=0 blocked=0 pending=12670 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s666:356s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__wikics__s888:240s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s888:1792s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:96%260808-11:21:25 (UTC+8)
593tasks total=14270 running=3 done=1589 failed=0 blocked=0 pending=12678 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s666:1205s;cuda2:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__wikics__s888:398s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s888:2240s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:100%260808-10:50:20 (UTC+8)
592tasks total=14270 running=3 done=1587 failed=0 blocked=0 pending=12680 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s666:459s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s888:1232s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s888:1494s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:100%260808-10:37:55 (UTC+8)
591tasks total=14270 running=3 done=1587 failed=0 blocked=0 pending=12680 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s666:278s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s888:1051s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s888:1313s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:96%260808-10:34:52 (UTC+8)
590tasks total=14270 running=3 done=1585 failed=0 blocked=0 pending=12682 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s666:659s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s888:499s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s888:761s | GPU cuda1:7851/24564MiB:95%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:96%260808-10:25:40 (UTC+8)
589tasks total=14270 running=3 done=1580 failed=0 blocked=0 pending=12687 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-a__llama3_8b__wikics__s666:484s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s888:970s;cuda3:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s888:1110s | GPU cuda1:7851/24564MiB:98%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:96%260808-10:07:18 (UTC+8)
588tasks total=14270 running=3 done=1578 failed=0 blocked=0 pending=12689 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s666:735s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s888:419s;cuda3:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s888:558s | GPU cuda1:7851/24564MiB:95%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260808-09:58:05 (UTC+8)
587tasks total=14270 running=3 done=1578 failed=0 blocked=0 pending=12689 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s666:367s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s888:52s;cuda3:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s888:191s | GPU cuda1:7851/24564MiB:98%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:97%260808-09:51:58 (UTC+8)
586tasks total=14270 running=3 done=1574 failed=0 blocked=0 pending=12693 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama3_8b__wikics__s666:429s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__wikics__s888:233s;cuda3:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s888:900s | GPU cuda1:7851/24564MiB:94%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:97%260808-09:39:39 (UTC+8)
585tasks total=14270 running=3 done=1571 failed=0 blocked=0 pending=12696 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s666:4618s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__wikics__s888:595s;cuda3:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s888:348s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:100%;cuda3:7851/24564MiB:97%260808-09:30:28 (UTC+8)
584tasks total=14270 running=3 done=1568 failed=0 blocked=0 pending=12699 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s666:4248s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__wikics__s888:225s;cuda3:trace_context:trace_bank__base__Random-256-b__llama3_8b__wikics__s888:1027s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:49%260808-09:24:18 (UTC+8)
583tasks total=14270 running=3 done=1564 failed=0 blocked=0 pending=12703 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s666:2960s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama31_4b__wikics__s888:231s;cuda3:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s888:783s | GPU cuda1:7851/24564MiB:0%;cuda2:4605/24564MiB:97%;cuda3:7851/24564MiB:96%260808-09:02:50 (UTC+8)
582tasks total=14270 running=3 done=1562 failed=0 blocked=0 pending=12705 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s666:2408s;cuda2:trace_context:trace_bank__n1__Random-128-b__llama31_4b__wikics__s888:192s;cuda3:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s888:231s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:95%260808-08:53:37 (UTC+8)
581tasks total=14270 running=3 done=1555 failed=0 blocked=0 pending=12712 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s666:734s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s888:599s;cuda3:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s888:296s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:94%;cuda3:7851/24564MiB:95%260808-08:25:43 (UTC+8)
580tasks total=14270 running=3 done=1544 failed=0 blocked=0 pending=12723 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s666:1618s;cuda2:trace_context:trace_bank__n1__Full-128-b__llama31_4b__wikics__s888:56s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s888:1003s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:95%260808-07:20:27 (UTC+8)
579tasks total=14270 running=3 done=1541 failed=0 blocked=0 pending=12726 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s666:877s;cuda2:trace_context:trace_bank__n1__Semantic-256-b__llama31_4b__wikics__s888:499s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s888:262s | GPU cuda1:7851/24564MiB:0%;cuda2:4605/24564MiB:100%;cuda3:7851/24564MiB:100%260808-07:08:05 (UTC+8)
578tasks total=14270 running=3 done=1537 failed=0 blocked=0 pending=12730 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s666:320s;cuda2:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__wikics__s888:615s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s888:751s | GPU cuda1:7851/24564MiB:96%;cuda2:4605/24564MiB:97%;cuda3:7851/24564MiB:95%260808-06:58:49 (UTC+8)
577tasks total=14270 running=3 done=1535 failed=0 blocked=0 pending=12732 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s666:3674s;cuda2:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__wikics__s888:64s;cuda3:trace_context:trace_bank__base__Full-256-b__llama3_8b__wikics__s888:199s | GPU cuda1:7851/24564MiB:0%;cuda2:4605/24564MiB:0%;cuda3:7851/24564MiB:97%260808-06:49:37 (UTC+8)
576tasks total=14270 running=3 done=1532 failed=0 blocked=0 pending=12735 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s666:3123s;cuda2:trace_context:trace_bank__n1__Full-512-b__llama31_4b__wikics__s888:428s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s888:689s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:100%;cuda3:7851/24564MiB:97%260808-06:40:25 (UTC+8)
575tasks total=14270 running=3 done=1528 failed=0 blocked=0 pending=12739 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s666:2382s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s888:601s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__wikics__s888:820s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:100%;cuda3:7851/24564MiB:95%260808-06:28:06 (UTC+8)
574tasks total=14270 running=3 done=1528 failed=0 blocked=0 pending=12739 | results done=595/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s666:1831s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s888:50s;cuda3:trace_context:trace_bank__base__Full-128-b__llama3_8b__wikics__s888:268s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:95%260808-06:18:54 (UTC+8)
573tasks total=14270 running=3 done=1492 failed=0 blocked=0 pending=12775 | results done=565/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s666:538s;cuda2:trace_downstream:run__C3_Diverse__llama31_4b__wikics__s888__gcn3:25s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s888:1038s | GPU cuda1:7851/24564MiB:0%;cuda2:1637/24564MiB:80%;cuda3:7851/24564MiB:97%260808-05:57:21 (UTC+8)
572tasks total=14270 running=3 done=1481 failed=0 blocked=0 pending=12786 | results done=554/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s666:172s;cuda2:trace_downstream:run__T9_Oracle__llama31_4b__wikics__s888__gcn3:22s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s888:672s | GPU cuda1:7851/24564MiB:0%;cuda2:1637/24564MiB:80%;cuda3:7851/24564MiB:94%260808-05:51:15 (UTC+8)
571tasks total=14270 running=3 done=1462 failed=0 blocked=0 pending=12805 | results done=536/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s666:3748s;cuda2:trace_downstream:run__T6_TRACEProxy__llama31_4b__wikics__s888__gcn1:0s;cuda3:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s888:306s | GPU cuda1:7851/24564MiB:97%;cuda2:14/24564MiB:0%;cuda3:7851/24564MiB:96%260808-05:45:08 (UTC+8)
570tasks total=14270 running=3 done=1448 failed=0 blocked=0 pending=12819 | results done=523/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s666:3011s;cuda2:trace_downstream:run__T3_TeacherOptGlobal__llama31_4b__wikics__s888__sage3:15s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s888:640s | GPU cuda1:7851/24564MiB:97%;cuda2:1441/24564MiB:73%;cuda3:7851/24564MiB:97%260808-05:32:51 (UTC+8)
569tasks total=14270 running=3 done=1430 failed=0 blocked=0 pending=12837 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s666:1718s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s888:341s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s888:926s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:97%260808-05:11:19 (UTC+8)
568tasks total=14270 running=3 done=1430 failed=0 blocked=0 pending=12837 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s666:1537s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s888:160s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s888:745s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:97%260808-05:08:18 (UTC+8)
567tasks total=14270 running=3 done=1429 failed=0 blocked=0 pending=12838 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s666:1356s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s888:1432s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s888:564s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260808-05:05:17 (UTC+8)
566tasks total=14270 running=3 done=1428 failed=0 blocked=0 pending=12839 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s666:614s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s888:690s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s888:1269s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260808-04:52:55 (UTC+8)
565tasks total=14270 running=3 done=1422 failed=0 blocked=0 pending=12845 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s666:2622s;cuda2:trace_context:trace_bank__base__HopQuota-512-b__llama31_4b__wikics__s888:80s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__wikics__s888:501s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:100%;cuda3:7851/24564MiB:96%260808-04:28:07 (UTC+8)
564tasks total=14270 running=3 done=1420 failed=0 blocked=0 pending=12847 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s666:1880s;cuda2:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s888:210s;cuda3:sft:sft__trace__llama3_8b__wikics__s888:3685s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:97%;cuda3:10479/24564MiB:100%260808-04:15:45 (UTC+8)
563tasks total=14270 running=3 done=1416 failed=0 blocked=0 pending=12851 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s666:950s;cuda2:trace_context:trace_bank__base__HopQuota-256-a__llama31_4b__wikics__s888:434s;cuda3:sft:sft__trace__llama3_8b__wikics__s888:2755s | GPU cuda1:7851/24564MiB:0%;cuda2:4605/24564MiB:97%;cuda3:10479/24564MiB:100%260808-04:00:16 (UTC+8)
562tasks total=14270 running=3 done=1408 failed=0 blocked=0 pending=12859 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s666:2722s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s888:3385s;cuda3:sft:sft__trace__llama3_8b__wikics__s888:1081s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:95%;cuda3:10477/24564MiB:99%260808-03:32:21 (UTC+8)
561tasks total=14270 running=3 done=1408 failed=0 blocked=0 pending=12859 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s666:2356s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s888:3018s;cuda3:sft:sft__trace__llama3_8b__wikics__s888:714s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:10477/24564MiB:99%260808-03:26:15 (UTC+8)
560tasks total=14270 running=3 done=1408 failed=0 blocked=0 pending=12859 | results done=512/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s666:1804s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s888:2467s;cuda3:sft:sft__trace__llama3_8b__wikics__s888:163s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:0%;cuda3:10477/24564MiB:100%260808-03:17:03 (UTC+8)
559tasks total=14270 running=3 done=1406 failed=0 blocked=0 pending=12861 | results done=511/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s666:1623s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s888:2286s;cuda3:external_downstream:run__LinguGKD__llama31_4b__wikics__s888__sage3:15s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:1849/24564MiB:31%260808-03:14:02 (UTC+8)
558tasks total=14270 running=3 done=1399 failed=0 blocked=0 pending=12868 | results done=506/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s666:1442s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s888:2105s;cuda3:lingu_bank:lingu_bank__llama31_4b__wikics__s888:1563s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:0%;cuda3:5545/24564MiB:95%260808-03:11:01 (UTC+8)
557tasks total=14270 running=3 done=1399 failed=0 blocked=0 pending=12868 | results done=506/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s666:148s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s888:811s;cuda3:lingu_bank:lingu_bank__llama31_4b__wikics__s888:268s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:5545/24564MiB:95%260808-02:49:27 (UTC+8)
556tasks total=14270 running=3 done=1397 failed=0 blocked=0 pending=12870 | results done=506/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s666:3227s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s888:630s;cuda3:lingu_bank:lingu_bank__llama31_4b__wikics__s888:87s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:0%;cuda3:5545/24564MiB:0%260808-02:46:26 (UTC+8)
555tasks total=14270 running=3 done=1395 failed=0 blocked=0 pending=12872 | results done=506/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s666:2306s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s888:3389s;cuda3:sft:sft__lingu__llama31_4b__wikics__s888:3945s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:0%;cuda3:8491/24564MiB:100%260808-02:31:05 (UTC+8)
554tasks total=14270 running=3 done=1395 failed=0 blocked=0 pending=12872 | results done=506/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s666:1010s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s888:2094s;cuda3:sft:sft__lingu__llama31_4b__wikics__s888:2650s | GPU cuda1:7851/24564MiB:95%;cuda2:5445/24564MiB:0%;cuda3:8491/24564MiB:100%260808-02:09:29 (UTC+8)
553tasks total=14270 running=3 done=1395 failed=0 blocked=0 pending=12872 | results done=506/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s666:459s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s888:1543s;cuda3:sft:sft__lingu__llama31_4b__wikics__s888:2098s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:0%;cuda3:8491/24564MiB:100%260808-02:00:18 (UTC+8)
552tasks total=14270 running=3 done=1394 failed=0 blocked=0 pending=12873 | results done=506/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s666:3018s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s888:800s;cuda3:sft:sft__lingu__llama31_4b__wikics__s888:1356s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:0%;cuda3:8491/24564MiB:99%260808-01:47:56 (UTC+8)
551tasks total=14270 running=3 done=1394 failed=0 blocked=0 pending=12873 | results done=506/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s666:2466s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s888:249s;cuda3:sft:sft__lingu__llama31_4b__wikics__s888:804s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:8491/24564MiB:100%260808-01:38:44 (UTC+8)
550tasks total=14270 running=3 done=1391 failed=0 blocked=0 pending=12876 | results done=506/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s666:1914s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s888:2746s;cuda3:sft:sft__lingu__llama31_4b__wikics__s888:252s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:0%;cuda3:8489/24564MiB:99%260808-01:29:32 (UTC+8)
549tasks total=14270 running=3 done=1386 failed=0 blocked=0 pending=12881 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s666:1172s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s888:2005s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s42:1818s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260808-01:17:10 (UTC+8)
548tasks total=14270 running=3 done=1386 failed=0 blocked=0 pending=12881 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s666:620s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s888:1453s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s42:1267s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260808-01:07:58 (UTC+8)
547tasks total=14270 running=3 done=1386 failed=0 blocked=0 pending=12881 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s666:438s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s888:1271s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s42:1085s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260808-01:04:57 (UTC+8)
546tasks total=14270 running=3 done=1386 failed=0 blocked=0 pending=12881 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s666:257s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s888:1089s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s42:903s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260808-01:01:55 (UTC+8)
545tasks total=14270 running=3 done=1384 failed=0 blocked=0 pending=12883 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s666:1871s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s888:347s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-b__llama3_8b__wikics__s42:161s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:100%;cuda3:7851/24564MiB:100%260808-00:49:33 (UTC+8)
544tasks total=14270 running=3 done=1382 failed=0 blocked=0 pending=12885 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s666:1313s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s888:2865s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s42:1807s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260808-00:40:15 (UTC+8)
543tasks total=14270 running=3 done=1382 failed=0 blocked=0 pending=12885 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s666:1132s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s888:2684s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s42:1626s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:96%260808-00:37:13 (UTC+8)
542tasks total=14270 running=3 done=1382 failed=0 blocked=0 pending=12885 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s666:951s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s888:2503s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s42:1445s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:96%260808-00:34:12 (UTC+8)
541tasks total=14270 running=3 done=1382 failed=0 blocked=0 pending=12885 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s666:770s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s888:2322s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s42:1264s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:95%260808-00:31:11 (UTC+8)
540tasks total=14270 running=3 done=1382 failed=0 blocked=0 pending=12885 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s666:401s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s888:1954s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s42:896s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:98%260808-00:25:04 (UTC+8)
539tasks total=14270 running=3 done=1381 failed=0 blocked=0 pending=12886 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s666:2204s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s888:1401s;cuda3:trace_context:trace_bank__n2__HopQuota-1024-a__llama3_8b__wikics__s42:343s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:98%260808-00:15:51 (UTC+8)
538tasks total=14270 running=3 done=1379 failed=0 blocked=0 pending=12888 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s666:1652s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s888:849s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s42:1162s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260808-00:06:38 (UTC+8)
537tasks total=14270 running=3 done=1379 failed=0 blocked=0 pending=12888 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s666:915s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s888:112s;cuda3:trace_context:trace_bank__n2__HopQuota-512-b__llama3_8b__wikics__s42:425s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:97%260807-23:54:21 (UTC+8)
536tasks total=14270 running=3 done=1375 failed=0 blocked=0 pending=12892 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s666:363s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s888:2369s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s42:1243s | GPU cuda1:7851/24564MiB:100%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:97%260807-23:45:09 (UTC+8)
535tasks total=14270 running=3 done=1373 failed=0 blocked=0 pending=12894 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s666:1110s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s888:1627s;cuda3:trace_context:trace_bank__n2__HopQuota-512-a__llama3_8b__wikics__s42:500s | GPU cuda1:7851/24564MiB:97%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:97%260807-23:32:48 (UTC+8)
534tasks total=14270 running=3 done=1371 failed=0 blocked=0 pending=12896 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s666:367s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s888:885s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s42:716s | GPU cuda1:7851/24564MiB:97%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:97%260807-23:20:24 (UTC+8)
533tasks total=14270 running=3 done=1370 failed=0 blocked=0 pending=12897 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s666:1303s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s888:333s;cuda3:trace_context:trace_bank__n2__HopQuota-256-b__llama3_8b__wikics__s42:164s | GPU cuda1:7851/24564MiB:70%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:95%260807-23:11:13 (UTC+8)
532tasks total=14270 running=3 done=1366 failed=0 blocked=0 pending=12901 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama3_8b__wikics__s666:1s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s888:1810s;cuda3:trace_context:trace_bank__n2__HopQuota-128-b__llama3_8b__wikics__s42:637s | GPU cuda1:14/24564MiB:0%;cuda2:4625/24564MiB:96%;cuda3:7851/24564MiB:96%260807-22:49:32 (UTC+8)
531tasks total=14270 running=3 done=1364 failed=0 blocked=0 pending=12903 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s666:724s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s888:1628s;cuda3:trace_context:trace_bank__n2__HopQuota-128-b__llama3_8b__wikics__s42:455s | GPU cuda1:7851/24564MiB:97%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:95%260807-22:46:29 (UTC+8)
530tasks total=14270 running=3 done=1363 failed=0 blocked=0 pending=12904 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s666:172s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s888:1076s;cuda3:trace_context:trace_bank__n2__HopQuota-128-a__llama3_8b__wikics__s42:720s | GPU cuda1:7851/24564MiB:0%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:97%260807-22:37:16 (UTC+8)
529tasks total=14270 running=3 done=1355 failed=0 blocked=0 pending=12912 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s666:2281s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s888:2564s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s42:4078s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:0%260807-22:18:34 (UTC+8)
528tasks total=14270 running=3 done=1355 failed=0 blocked=0 pending=12912 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s666:1915s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s888:2198s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s42:3712s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260807-22:12:27 (UTC+8)
527tasks total=14270 running=3 done=1355 failed=0 blocked=0 pending=12912 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s666:987s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s888:1270s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s42:2784s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260807-21:57:00 (UTC+8)
526tasks total=14270 running=3 done=1355 failed=0 blocked=0 pending=12912 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s666:431s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s888:713s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s42:2227s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:0%260807-21:47:43 (UTC+8)
525tasks total=14270 running=3 done=1353 failed=0 blocked=0 pending=12914 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s666:2015s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s888:2589s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s42:1485s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:0%260807-21:35:20 (UTC+8)
524tasks total=14270 running=3 done=1353 failed=0 blocked=0 pending=12914 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s666:1270s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s888:1844s;cuda3:trace_context:trace_bank__n2__Diverse-1024-b__llama3_8b__wikics__s42:740s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260807-21:22:55 (UTC+8)
523tasks total=14270 running=3 done=1352 failed=0 blocked=0 pending=12915 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s666:527s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s888:1101s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s42:4424s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:96%260807-21:10:33 (UTC+8)
522tasks total=14270 running=3 done=1352 failed=0 blocked=0 pending=12915 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s666:343s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s888:917s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s42:4241s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:100%260807-21:07:31 (UTC+8)
521tasks total=14270 running=3 done=1350 failed=0 blocked=0 pending=12917 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s666:1254s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s888:365s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s42:3688s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:0%260807-20:58:17 (UTC+8)
520tasks total=14270 running=3 done=1347 failed=0 blocked=0 pending=12920 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-512-b__llama3_8b__wikics__s666:699s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s888:1405s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s42:3133s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:95%260807-20:49:05 (UTC+8)
519tasks total=14270 running=3 done=1343 failed=0 blocked=0 pending=12924 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-b__llama3_8b__wikics__s666:794s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s888:572s;cuda3:trace_context:trace_bank__n2__Diverse-1024-a__llama3_8b__wikics__s42:711s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260807-20:08:42 (UTC+8)
518tasks total=14270 running=3 done=1337 failed=0 blocked=0 pending=12930 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s666:731s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s888:397s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s42:3160s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260807-19:50:01 (UTC+8)
517tasks total=14270 running=3 done=1336 failed=0 blocked=0 pending=12931 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s666:179s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s888:790s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s42:2608s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260807-19:40:48 (UTC+8)
516tasks total=14270 running=3 done=1334 failed=0 blocked=0 pending=12933 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s666:317s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s888:48s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s42:1866s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:24%;cuda3:7851/24564MiB:97%260807-19:28:26 (UTC+8)
515tasks total=14270 running=3 done=1331 failed=0 blocked=0 pending=12936 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s666:136s;cuda2:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__wikics__s888:406s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s42:1685s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:94%;cuda3:7851/24564MiB:0%260807-19:25:25 (UTC+8)
514tasks total=14270 running=3 done=1330 failed=0 blocked=0 pending=12937 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s666:830s;cuda2:trace_context:trace_bank__base__Semantic-128-b__llama31_4b__wikics__s888:224s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s42:1503s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:0%260807-19:22:23 (UTC+8)
513tasks total=14270 running=3 done=1329 failed=0 blocked=0 pending=12938 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-128-a__llama3_8b__wikics__s666:88s;cuda2:trace_context:trace_bank__base__Semantic-128-a__llama31_4b__wikics__s888:15s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s42:761s | GPU cuda1:7851/24564MiB:98%;cuda2:4549/24564MiB:0%;cuda3:7851/24564MiB:0%260807-19:10:01 (UTC+8)
512tasks total=14270 running=3 done=1324 failed=0 blocked=0 pending=12943 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s666:1865s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s888:1047s;cuda3:trace_context:trace_bank__n2__Diverse-512-b__llama3_8b__wikics__s42:209s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:100%260807-19:00:49 (UTC+8)
511tasks total=14270 running=3 done=1323 failed=0 blocked=0 pending=12944 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s666:1314s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s888:496s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s42:3217s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:97%260807-18:51:38 (UTC+8)
510tasks total=14270 running=3 done=1323 failed=0 blocked=0 pending=12944 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s666:948s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s888:130s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s42:2850s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:94%;cuda3:7851/24564MiB:0%260807-18:45:32 (UTC+8)
509tasks total=14270 running=3 done=1321 failed=0 blocked=0 pending=12946 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s666:1794s;cuda2:trace_context:trace_bank__base__Random-1024-a__llama31_4b__wikics__s888:230s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s42:1370s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:94%;cuda3:7851/24564MiB:97%260807-18:20:51 (UTC+8)
508tasks total=14270 running=3 done=1318 failed=0 blocked=0 pending=12949 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s666:690s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s888:50s;cuda3:trace_context:trace_bank__n2__Diverse-512-a__llama3_8b__wikics__s42:266s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260807-18:02:30 (UTC+8)
507tasks total=14270 running=3 done=1315 failed=0 blocked=0 pending=12952 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s666:323s;cuda2:trace_context:trace_bank__base__Random-512-a__llama31_4b__wikics__s888:600s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s42:3086s | GPU cuda1:7851/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260807-17:56:21 (UTC+8)
506tasks total=14270 running=3 done=1310 failed=0 blocked=0 pending=12957 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s666:450s;cuda2:trace_context:trace_bank__base__Random-256-b__llama31_4b__wikics__s888:320s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s42:2156s | GPU cuda1:7851/24564MiB:96%;cuda2:4625/24564MiB:78%;cuda3:7851/24564MiB:0%260807-17:40:51 (UTC+8)
505tasks total=14270 running=3 done=1308 failed=0 blocked=0 pending=12959 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s666:951s;cuda2:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s888:413s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s42:1602s | GPU cuda1:7851/24564MiB:97%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:94%260807-17:31:37 (UTC+8)
504tasks total=14270 running=3 done=1308 failed=0 blocked=0 pending=12959 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s666:580s;cuda2:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s888:41s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s42:1231s | GPU cuda1:7851/24564MiB:98%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:97%260807-17:25:26 (UTC+8)
503tasks total=14270 running=3 done=1305 failed=0 blocked=0 pending=12962 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s666:399s;cuda2:trace_context:trace_bank__base__Random-128-b__llama31_4b__wikics__s888:373s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s42:1050s | GPU cuda1:7851/24564MiB:97%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:97%260807-17:22:25 (UTC+8)
502tasks total=14270 running=3 done=1305 failed=0 blocked=0 pending=12962 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-256-a__llama3_8b__wikics__s666:33s;cuda2:trace_context:trace_bank__base__Random-128-b__llama31_4b__wikics__s888:7s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s42:684s | GPU cuda1:7851/24564MiB:98%;cuda2:4405/24564MiB:23%;cuda3:7851/24564MiB:0%260807-17:16:19 (UTC+8)
501tasks total=14270 running=3 done=1302 failed=0 blocked=0 pending=12965 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s666:543s;cuda2:trace_context:trace_bank__base__Random-128-a__llama31_4b__wikics__s888:146s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s42:315s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:0%260807-17:10:10 (UTC+8)
500tasks total=14270 running=3 done=1299 failed=0 blocked=0 pending=12968 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s666:362s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s888:1540s;cuda3:trace_context:trace_bank__n2__Diverse-256-b__llama3_8b__wikics__s42:134s | GPU cuda1:7851/24564MiB:95%;cuda2:5445/24564MiB:94%;cuda3:7851/24564MiB:0%260807-17:07:09 (UTC+8)
499tasks total=14270 running=3 done=1297 failed=0 blocked=0 pending=12970 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s666:687s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s888:989s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s42:2788s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:0%260807-16:57:58 (UTC+8)
498tasks total=14270 running=3 done=1297 failed=0 blocked=0 pending=12970 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s666:506s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s888:808s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s42:2607s | GPU cuda1:7851/24564MiB:98%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260807-16:54:57 (UTC+8)
497tasks total=14270 running=3 done=1295 failed=0 blocked=0 pending=12972 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-b__llama3_8b__wikics__s666:1034s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s888:249s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s42:2048s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260807-16:45:38 (UTC+8)
496tasks total=14270 running=3 done=1294 failed=0 blocked=0 pending=12973 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-b__llama3_8b__wikics__s666:668s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s888:1462s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s42:1682s | GPU cuda1:7851/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260807-16:39:32 (UTC+8)
495tasks total=14270 running=3 done=1293 failed=0 blocked=0 pending=12974 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s666:1015s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s888:723s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s42:943s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260807-16:27:13 (UTC+8)
494tasks total=14270 running=3 done=1293 failed=0 blocked=0 pending=12974 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s666:648s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s888:356s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s42:576s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260807-16:21:06 (UTC+8)
493tasks total=14270 running=3 done=1290 failed=0 blocked=0 pending=12977 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s666:97s;cuda2:trace_context:trace_bank__base__Full-256-b__llama31_4b__wikics__s888:458s;cuda3:trace_context:trace_bank__n2__Diverse-256-a__llama3_8b__wikics__s42:25s | GPU cuda1:7851/24564MiB:97%;cuda2:4625/24564MiB:88%;cuda3:7835/24564MiB:0%260807-16:11:55 (UTC+8)
492tasks total=14270 running=3 done=1285 failed=0 blocked=0 pending=12982 | results done=505/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s666:807s;cuda2:trace_context:trace_bank__base__Full-256-a__llama31_4b__wikics__s888:353s;cuda3:trace_context:trace_bank__n2__Diverse-128-b__llama3_8b__wikics__s42:2234s | GPU cuda1:7851/24564MiB:97%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:0%260807-15:59:23 (UTC+8)
491tasks total=14270 running=3 done=1268 failed=0 blocked=0 pending=12999 | results done=498/5050 | RUNNING cuda1:trace_downstream:run__G4_AmortizedOneShot__llama3_8b__wikics__s666__gcn3:17s;cuda2:trace_context:trace_bank__base__Semantic-256-a__llama31_4b__wikics__s888:541s;cuda3:trace_context:trace_bank__n2__Diverse-128-a__llama3_8b__wikics__s42:2928s | GPU cuda1:1637/24564MiB:80%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:0%260807-15:21:25 (UTC+8)
490tasks total=14270 running=3 done=1184 failed=0 blocked=0 pending=13083 | results done=424/5050 | RUNNING cuda1:trace_downstream:run__T3_TeacherOptGlobal__llama3_8b__wikics__s666__gcn3:16s;cuda2:sft:sft__trace__llama31_4b__wikics__s888:2538s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s42:2229s | GPU cuda1:1635/24564MiB:78%;cuda2:7969/24564MiB:30%;cuda3:7851/24564MiB:43%260807-14:30:40 (UTC+8)
489tasks total=14270 running=3 done=1178 failed=0 blocked=0 pending=13089 | results done=422/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s666:561s;cuda2:sft:sft__trace__llama31_4b__wikics__s888:826s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s42:517s | GPU cuda1:7851/24564MiB:0%;cuda2:7967/24564MiB:99%;cuda3:7851/24564MiB:96%260807-14:02:12 (UTC+8)
488tasks total=14270 running=3 done=1178 failed=0 blocked=0 pending=13089 | results done=422/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s666:188s;cuda2:sft:sft__trace__llama31_4b__wikics__s888:453s;cuda3:trace_context:trace_bank__n2__Semantic-1024-b__llama3_8b__wikics__s42:144s | GPU cuda1:7851/24564MiB:96%;cuda2:7967/24564MiB:99%;cuda3:7851/24564MiB:100%260807-13:55:55 (UTC+8)
487tasks total=14270 running=3 done=1177 failed=0 blocked=0 pending=13090 | results done=422/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s666:7s;cuda2:sft:sft__trace__llama31_4b__wikics__s888:271s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s42:2305s | GPU cuda1:7561/24564MiB:0%;cuda2:7967/24564MiB:67%;cuda3:7851/24564MiB:23%260807-13:52:53 (UTC+8)
486tasks total=14270 running=3 done=1164 failed=0 blocked=0 pending=13103 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s666:1226s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s666:554s;cuda3:trace_context:trace_bank__n2__Semantic-1024-a__llama3_8b__wikics__s42:27s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:100%260807-13:14:55 (UTC+8)
485tasks total=14270 running=3 done=1162 failed=0 blocked=0 pending=13105 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s666:1044s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s666:372s;cuda3:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s42:1327s | GPU cuda1:7851/24564MiB:97%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:97%260807-13:11:53 (UTC+8)
484tasks total=14270 running=3 done=1161 failed=0 blocked=0 pending=13106 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s666:669s;cuda2:sft:sft__lingu__llama3_8b__wikics__s666:6598s;cuda3:trace_context:trace_bank__n2__Semantic-512-b__llama3_8b__wikics__s42:952s | GPU cuda1:7851/24564MiB:97%;cuda2:11839/24564MiB:76%;cuda3:7851/24564MiB:97%260807-13:05:41 (UTC+8)
483tasks total=14270 running=3 done=1159 failed=0 blocked=0 pending=13108 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s666:677s;cuda2:sft:sft__lingu__llama3_8b__wikics__s666:5266s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s42:1100s | GPU cuda1:7851/24564MiB:97%;cuda2:11839/24564MiB:98%;cuda3:7851/24564MiB:100%260807-12:43:34 (UTC+8)
482tasks total=14270 running=3 done=1159 failed=0 blocked=0 pending=13108 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s666:107s;cuda2:sft:sft__lingu__llama3_8b__wikics__s666:4696s;cuda3:trace_context:trace_bank__n2__Semantic-512-a__llama3_8b__wikics__s42:530s | GPU cuda1:7851/24564MiB:100%;cuda2:11839/24564MiB:99%;cuda3:7851/24564MiB:97%260807-12:33:59 (UTC+8)
481tasks total=14270 running=3 done=1153 failed=0 blocked=0 pending=13114 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s666:805s;cuda2:sft:sft__lingu__llama3_8b__wikics__s666:3557s;cuda3:trace_context:trace_bank__n2__Semantic-128-b__llama3_8b__wikics__s42:288s | GPU cuda1:7851/24564MiB:97%;cuda2:11839/24564MiB:99%;cuda3:7851/24564MiB:95%260807-12:14:58 (UTC+8)
480tasks total=14270 running=3 done=1152 failed=0 blocked=0 pending=13115 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s666:236s;cuda2:sft:sft__lingu__llama3_8b__wikics__s666:2988s;cuda3:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s42:616s | GPU cuda1:7851/24564MiB:97%;cuda2:11839/24564MiB:100%;cuda3:7851/24564MiB:97%260807-12:05:33 (UTC+8)
479tasks total=14270 running=3 done=1152 failed=0 blocked=0 pending=13115 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s666:55s;cuda2:sft:sft__lingu__llama3_8b__wikics__s666:2806s;cuda3:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s42:434s | GPU cuda1:7851/24564MiB:0%;cuda2:11839/24564MiB:100%;cuda3:7851/24564MiB:96%260807-12:02:27 (UTC+8)
478tasks total=14270 running=3 done=1149 failed=0 blocked=0 pending=13118 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s666:664s;cuda2:sft:sft__lingu__llama3_8b__wikics__s666:2624s;cuda3:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s42:252s | GPU cuda1:7851/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:7851/24564MiB:98%260807-11:59:25 (UTC+8)
477tasks total=14270 running=3 done=1149 failed=0 blocked=0 pending=13118 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s666:483s;cuda2:sft:sft__lingu__llama3_8b__wikics__s666:2442s;cuda3:trace_context:trace_bank__n2__Semantic-128-a__llama3_8b__wikics__s42:70s | GPU cuda1:7851/24564MiB:98%;cuda2:11839/24564MiB:99%;cuda3:7851/24564MiB:95%260807-11:56:23 (UTC+8)
476tasks total=14270 running=3 done=1147 failed=0 blocked=0 pending=13120 | results done=416/5050 | RUNNING cuda1:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s666:302s;cuda2:sft:sft__lingu__llama3_8b__wikics__s666:2261s;cuda3:trace_context:trace_bank__n2__Random-1024-b__llama3_8b__wikics__s42:2208s | GPU cuda1:7851/24564MiB:95%;cuda2:11839/24564MiB:100%;cuda3:7851/24564MiB:100%260807-11:53:21 (UTC+8)
475tasks total=14270 running=3 done=1134 failed=0 blocked=0 pending=13133 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s666:838s;cuda2:trace_context:trace_bank__n2__HopQuota-1024-a__llama31_4b__wikics__s666:576s;cuda3:trace_context:trace_bank__n2__Random-512-b__llama3_8b__wikics__s42:1225s | GPU cuda1:7851/24564MiB:100%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:97%260807-10:34:07 (UTC+8)
474tasks total=14270 running=3 done=1122 failed=0 blocked=0 pending=13145 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s666:1660s;cuda2:trace_context:trace_bank__n2__HopQuota-128-b__llama31_4b__wikics__s666:444s;cuda3:trace_context:trace_bank__n2__Random-256-b__llama3_8b__wikics__s42:124s | GPU cuda1:7851/24564MiB:95%;cuda2:4565/24564MiB:95%;cuda3:7851/24564MiB:95%260807-09:33:55 (UTC+8)
473tasks total=14270 running=3 done=1112 failed=0 blocked=0 pending=13155 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s666:3130s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s666:2510s;cuda3:trace_context:trace_bank__n2__Random-128-a__llama3_8b__wikics__s42:834s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:97%260807-08:59:05 (UTC+8)
472tasks total=14270 running=3 done=1110 failed=0 blocked=0 pending=13157 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s666:1991s;cuda2:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s666:1371s;cuda3:trace_context:trace_bank__n2__Full-1024-b__llama3_8b__wikics__s42:2013s | GPU cuda1:7851/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:98%260807-08:40:07 (UTC+8)
471tasks total=14270 running=3 done=1103 failed=0 blocked=0 pending=13164 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s666:423s;cuda2:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s666:2895s;cuda3:trace_context:trace_bank__n2__Full-256-b__llama3_8b__wikics__s42:252s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:97%260807-07:14:27 (UTC+8)
470tasks total=14270 running=3 done=1092 failed=0 blocked=0 pending=13175 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s666:2909s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s666:2092s;cuda3:trace_context:trace_bank__n2__Semantic-256-b__llama3_8b__wikics__s42:331s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260807-06:11:05 (UTC+8)
469tasks total=14270 running=3 done=1091 failed=0 blocked=0 pending=13176 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s666:2155s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s666:1338s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s42:658s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260807-05:58:32 (UTC+8)
468tasks total=14270 running=3 done=1091 failed=0 blocked=0 pending=13176 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s666:1973s;cuda2:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s666:1156s;cuda3:trace_context:trace_bank__n2__Semantic-256-a__llama3_8b__wikics__s42:476s | GPU cuda1:7851/24564MiB:0%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:28%260807-05:55:29 (UTC+8)
467tasks total=14270 running=3 done=1083 failed=0 blocked=0 pending=13184 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s666:2879s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s666:1808s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s42:1298s | GPU cuda1:7851/24564MiB:0%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:97%260807-05:20:43 (UTC+8)
466tasks total=14270 running=3 done=1083 failed=0 blocked=0 pending=13184 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s666:2505s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s666:1434s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s42:924s | GPU cuda1:7851/24564MiB:0%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:100%260807-05:14:30 (UTC+8)
465tasks total=14270 running=3 done=1083 failed=0 blocked=0 pending=13184 | results done=415/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s666:2131s;cuda2:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s666:1059s;cuda3:trace_context:trace_bank__n2__Full-512-a__llama3_8b__wikics__s42:550s | GPU cuda1:7851/24564MiB:0%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:100%260807-05:08:15 (UTC+8)
464tasks total=14270 running=3 done=1078 failed=0 blocked=0 pending=13189 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s666:805s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s666:2509s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s42:1484s | GPU cuda1:7851/24564MiB:0%;cuda2:4625/24564MiB:100%;cuda3:7851/24564MiB:96%260807-04:46:09 (UTC+8)
463tasks total=14270 running=3 done=1078 failed=0 blocked=0 pending=13189 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s666:232s;cuda2:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s666:1937s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-b__llama3_8b__wikics__s42:911s | GPU cuda1:7851/24564MiB:0%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:100%260807-04:36:45 (UTC+8)
462tasks total=14270 running=3 done=1074 failed=0 blocked=0 pending=13193 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s666:381s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s666:1711s;cuda3:trace_context:trace_bank__n1__HopQuota-1024-a__llama3_8b__wikics__s42:221s | GPU cuda1:7851/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:96%260807-03:49:07 (UTC+8)
461tasks total=14270 running=3 done=1072 failed=0 blocked=0 pending=13195 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s666:9s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s666:1339s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s42:1201s | GPU cuda1:7643/24564MiB:22%;cuda2:4565/24564MiB:32%;cuda3:7851/24564MiB:97%260807-03:42:53 (UTC+8)
460tasks total=14270 running=3 done=1069 failed=0 blocked=0 pending=13198 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s666:2177s;cuda2:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s666:1156s;cuda3:trace_context:trace_bank__n1__HopQuota-512-b__llama3_8b__wikics__s42:1018s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:97%260807-03:39:51 (UTC+8)
459tasks total=14270 running=3 done=1066 failed=0 blocked=0 pending=13201 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s666:2243s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s666:1495s;cuda3:trace_context:trace_bank__n1__HopQuota-512-a__llama3_8b__wikics__s42:89s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:98%260807-03:01:53 (UTC+8)
458tasks total=14270 running=3 done=1064 failed=0 blocked=0 pending=13203 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s666:1872s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s666:1124s;cuda3:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s42:646s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:97%260807-02:55:37 (UTC+8)
457tasks total=14270 running=3 done=1064 failed=0 blocked=0 pending=13203 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s666:1689s;cuda2:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s666:941s;cuda3:trace_context:trace_bank__n1__HopQuota-256-b__llama3_8b__wikics__s42:463s | GPU cuda1:7851/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:98%260807-02:52:36 (UTC+8)
456tasks total=14270 running=3 done=1059 failed=0 blocked=0 pending=13208 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s666:173s;cuda2:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s666:1046s;cuda3:trace_context:trace_bank__n1__HopQuota-128-b__llama3_8b__wikics__s42:672s | GPU cuda1:7851/24564MiB:100%;cuda2:5465/24564MiB:100%;cuda3:7851/24564MiB:98%260807-02:27:19 (UTC+8)
455tasks total=14270 running=3 done=1044 failed=0 blocked=0 pending=13223 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-128-a__llama3_8b__wikics__s666:757s;cuda2:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s666:353s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s42:2053s | GPU cuda1:7851/24564MiB:95%;cuda2:4885/24564MiB:100%;cuda3:7851/24564MiB:0%260807-01:17:37 (UTC+8)
454tasks total=14270 running=3 done=1036 failed=0 blocked=0 pending=13231 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s666:1370s;cuda2:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s666:1276s;cuda3:trace_context:trace_bank__n1__Diverse-1024-b__llama3_8b__wikics__s42:341s | GPU cuda1:7851/24564MiB:96%;cuda2:5465/24564MiB:95%;cuda3:7851/24564MiB:0%260807-00:49:05 (UTC+8)
453tasks total=14270 running=3 done=1034 failed=0 blocked=0 pending=13233 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s666:40s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s666:1511s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s42:3723s | GPU cuda1:7851/24564MiB:96%;cuda2:5465/24564MiB:100%;cuda3:7851/24564MiB:0%260807-00:26:56 (UTC+8)
452tasks total=14270 running=3 done=1033 failed=0 blocked=0 pending=13234 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s666:1987s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s666:1138s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s42:3349s | GPU cuda1:7851/24564MiB:95%;cuda2:5465/24564MiB:100%;cuda3:7851/24564MiB:100%260807-00:20:42 (UTC+8)
451tasks total=14270 running=3 done=1033 failed=0 blocked=0 pending=13234 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s666:1805s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s666:956s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s42:3167s | GPU cuda1:7851/24564MiB:100%;cuda2:5465/24564MiB:100%;cuda3:7851/24564MiB:0%260807-00:17:39 (UTC+8)
450tasks total=14270 running=3 done=1033 failed=0 blocked=0 pending=13234 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s666:855s;cuda2:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s666:6s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s42:2218s | GPU cuda1:7851/24564MiB:96%;cuda2:401/24564MiB:0%;cuda3:7851/24564MiB:96%260807-00:01:56 (UTC+8)
449tasks total=14270 running=3 done=1027 failed=0 blocked=0 pending=13240 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s666:794s;cuda2:trace_context:trace_bank__n2__Random-512-a__llama31_4b__wikics__s666:308s;cuda3:trace_context:trace_bank__n1__Diverse-1024-a__llama3_8b__wikics__s42:697s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:95%;cuda3:7851/24564MiB:0%260806-23:36:33 (UTC+8)
448tasks total=14270 running=3 done=1012 failed=0 blocked=0 pending=13255 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s666:737s;cuda2:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s666:564s;cuda3:trace_context:trace_bank__n1__Diverse-512-b__llama3_8b__wikics__s42:908s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:0%260806-22:36:16 (UTC+8)
447tasks total=14270 running=3 done=1006 failed=0 blocked=0 pending=13261 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s666:392s;cuda2:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s666:36s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s42:2714s | GPU cuda1:7851/24564MiB:95%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:97%260806-22:01:27 (UTC+8)
446tasks total=14270 running=3 done=1004 failed=0 blocked=0 pending=13263 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Random-128-a__llama3_8b__wikics__s666:200s;cuda2:trace_context:trace_bank__n2__Full-256-b__llama31_4b__wikics__s666:487s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s42:2522s | GPU cuda1:7851/24564MiB:95%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:0%260806-21:58:22 (UTC+8)
445tasks total=14270 running=3 done=997 failed=0 blocked=0 pending=13270 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s666:817s;cuda2:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s666:440s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s42:816s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:95%;cuda3:7851/24564MiB:0%260806-21:29:46 (UTC+8)
444tasks total=14270 running=3 done=997 failed=0 blocked=0 pending=13270 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s666:633s;cuda2:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s666:255s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s42:632s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:0%260806-21:26:45 (UTC+8)
443tasks total=14270 running=3 done=997 failed=0 blocked=0 pending=13270 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s666:450s;cuda2:trace_context:trace_bank__n2__Full-128-a__llama31_4b__wikics__s666:73s;cuda3:trace_context:trace_bank__n1__Diverse-512-a__llama3_8b__wikics__s42:449s | GPU cuda1:7851/24564MiB:96%;cuda2:4565/24564MiB:95%;cuda3:7851/24564MiB:0%260806-21:23:40 (UTC+8)
442tasks total=14270 running=3 done=989 failed=0 blocked=0 pending=13278 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s666:1060s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s666:610s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s42:2236s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260806-20:55:11 (UTC+8)
441tasks total=14270 running=3 done=989 failed=0 blocked=0 pending=13278 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s666:688s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s666:237s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s42:1863s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260806-20:48:57 (UTC+8)
440tasks total=14270 running=3 done=989 failed=0 blocked=0 pending=13278 | results done=414/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s666:506s;cuda2:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s666:55s;cuda3:trace_context:trace_bank__n1__Diverse-256-b__llama3_8b__wikics__s42:1681s | GPU cuda1:7851/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260806-20:45:55 (UTC+8)
439tasks total=14270 running=3 done=979 failed=0 blocked=0 pending=13288 | results done=413/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s666:893s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s666:816s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s42:3433s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:33%260806-20:17:28 (UTC+8)
438tasks total=14270 running=3 done=979 failed=0 blocked=0 pending=13288 | results done=413/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s666:710s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s666:633s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s42:3250s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:97%260806-20:14:23 (UTC+8)
437tasks total=14270 running=3 done=979 failed=0 blocked=0 pending=13288 | results done=413/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s666:524s;cuda2:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s666:448s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s42:3064s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:97%260806-20:11:20 (UTC+8)
436tasks total=14270 running=3 done=972 failed=0 blocked=0 pending=13295 | results done=413/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s666:185s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s666:711s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s42:978s | GPU cuda1:7851/24564MiB:98%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260806-19:36:30 (UTC+8)
435tasks total=14270 running=3 done=972 failed=0 blocked=0 pending=13295 | results done=413/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s666:2s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s666:527s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s42:794s | GPU cuda1:401/24564MiB:0%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260806-19:33:28 (UTC+8)
434tasks total=14270 running=3 done=963 failed=0 blocked=0 pending=13304 | results done=407/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s666:1012s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s666:345s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s42:613s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:97%;cuda3:7851/24564MiB:0%260806-19:30:24 (UTC+8)
433tasks total=14270 running=3 done=963 failed=0 blocked=0 pending=13304 | results done=407/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s666:829s;cuda2:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s666:162s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s42:429s | GPU cuda1:7851/24564MiB:14%;cuda2:4885/24564MiB:97%;cuda3:7851/24564MiB:98%260806-19:27:23 (UTC+8)
432tasks total=14270 running=3 done=962 failed=0 blocked=0 pending=13305 | results done=407/5050 | RUNNING cuda1:trace_context:trace_bank__base__Semantic-256-b__llama3_8b__wikics__s666:647s;cuda2:trace_context:trace_bank__n1__HopQuota-512-a__llama31_4b__wikics__s666:863s;cuda3:trace_context:trace_bank__n1__Diverse-256-a__llama3_8b__wikics__s42:247s | GPU cuda1:7851/24564MiB:97%;cuda2:4885/24564MiB:97%;cuda3:7851/24564MiB:97%260806-19:24:19 (UTC+8)
431tasks total=14270 running=3 done=942 failed=0 blocked=0 pending=13325 | results done=401/5050 | RUNNING cuda1:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s666:233s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s666:3898s;cuda3:trace_context:trace_bank__n1__Diverse-128-b__llama3_8b__wikics__s42:506s | GPU cuda1:7851/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260806-18:33:39 (UTC+8)
430tasks total=14270 running=3 done=939 failed=0 blocked=0 pending=13328 | results done=401/5050 | RUNNING cuda1:trace_context:trace_bank__base__Self__llama3_8b__wikics__s666:515s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s666:2002s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s42:1905s | GPU cuda1:7851/24564MiB:95%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:96%260806-18:01:55 (UTC+8)
429tasks total=14270 running=3 done=939 failed=0 blocked=0 pending=13328 | results done=401/5050 | RUNNING cuda1:trace_context:trace_bank__base__Self__llama3_8b__wikics__s666:333s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s666:1821s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s42:1724s | GPU cuda1:7851/24564MiB:95%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260806-17:58:54 (UTC+8)
428tasks total=14270 running=3 done=939 failed=0 blocked=0 pending=13328 | results done=401/5050 | RUNNING cuda1:trace_context:trace_bank__base__Self__llama3_8b__wikics__s666:151s;cuda2:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s666:1638s;cuda3:trace_context:trace_bank__n1__Diverse-128-a__llama3_8b__wikics__s42:1541s | GPU cuda1:7851/24564MiB:90%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:0%260806-17:55:52 (UTC+8)
427tasks total=14270 running=3 done=935 failed=0 blocked=0 pending=13332 | results done=401/5050 | RUNNING cuda1:sft:sft__trace__llama3_8b__wikics__s666:1227s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s666:2797s;cuda3:trace_context:trace_bank__n1__Semantic-1024-b__llama3_8b__wikics__s42:1040s | GPU cuda1:10477/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:95%260806-17:08:20 (UTC+8)
426tasks total=14270 running=3 done=925 failed=0 blocked=0 pending=13342 | results done=395/5050 | RUNNING cuda1:lingu_bank:lingu_bank__llama31_4b__wikics__s666:912s;cuda2:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s666:706s;cuda3:trace_context:trace_bank__n1__Semantic-1024-a__llama3_8b__wikics__s42:1296s | GPU cuda1:5545/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:100%260806-16:33:31 (UTC+8)
425tasks total=14270 running=3 done=920 failed=0 blocked=0 pending=13347 | results done=395/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s666:3993s;cuda2:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s666:2331s;cuda3:trace_context:trace_bank__n1__Semantic-512-b__llama3_8b__wikics__s42:1072s | GPU cuda1:8491/24564MiB:99%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260806-16:05:00 (UTC+8)
424tasks total=14270 running=3 done=916 failed=0 blocked=0 pending=13351 | results done=395/5050 | RUNNING cuda1:sft:sft__lingu__llama31_4b__wikics__s666:1145s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s666:2784s;cuda3:trace_context:trace_bank__n1__Semantic-128-b__llama3_8b__wikics__s42:605s | GPU cuda1:8489/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:95%260806-15:17:33 (UTC+8)
423tasks total=14270 running=3 done=908 failed=0 blocked=0 pending=13359 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-1024-b__llama31_4b__wikics__s42:63s;cuda2:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s666:125s;cuda3:trace_context:trace_bank__n1__Random-1024-b__llama3_8b__wikics__s42:1166s | GPU cuda1:5445/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:96%260806-14:33:15 (UTC+8)
422tasks total=14270 running=3 done=902 failed=0 blocked=0 pending=13365 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-512-b__llama31_4b__wikics__s42:142s;cuda2:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s666:896s;cuda3:trace_context:trace_bank__n1__Random-1024-a__llama3_8b__wikics__s42:1205s | GPU cuda1:4885/24564MiB:29%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:100%260806-13:55:16 (UTC+8)
421tasks total=14270 running=3 done=891 failed=0 blocked=0 pending=13376 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__HopQuota-128-a__llama31_4b__wikics__s42:320s;cuda2:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s666:1186s;cuda3:trace_context:trace_bank__n1__Random-512-a__llama3_8b__wikics__s42:1267s | GPU cuda1:4565/24564MiB:97%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:97%260806-13:07:47 (UTC+8)
420tasks total=14270 running=3 done=884 failed=0 blocked=0 pending=13383 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s42:1497s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s666:1618s;cuda3:trace_context:trace_bank__n1__Random-256-a__llama3_8b__wikics__s42:902s | GPU cuda1:5465/24564MiB:0%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:97%260806-12:26:40 (UTC+8)
419tasks total=14270 running=3 done=882 failed=0 blocked=0 pending=13385 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-b__llama31_4b__wikics__s42:361s;cuda2:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s666:481s;cuda3:trace_context:trace_bank__n1__Random-128-b__llama3_8b__wikics__s42:640s | GPU cuda1:5465/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:98%260806-12:07:41 (UTC+8)
418tasks total=14270 running=3 done=877 failed=0 blocked=0 pending=13390 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s42:2482s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s666:1940s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s42:2311s | GPU cuda1:5465/24564MiB:0%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:95%260806-11:42:24 (UTC+8)
417tasks total=14270 running=3 done=877 failed=0 blocked=0 pending=13390 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-1024-a__llama31_4b__wikics__s42:1157s;cuda2:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s666:615s;cuda3:trace_context:trace_bank__n1__Full-1024-b__llama3_8b__wikics__s42:986s | GPU cuda1:5465/24564MiB:100%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:100%260806-11:20:12 (UTC+8)
416tasks total=14270 running=3 done=871 failed=0 blocked=0 pending=13396 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s42:1761s;cuda2:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s666:1362s;cuda3:trace_context:trace_bank__n1__Full-1024-a__llama3_8b__wikics__s42:836s | GPU cuda1:4885/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:100%260806-10:39:06 (UTC+8)
415tasks total=14270 running=3 done=867 failed=0 blocked=0 pending=13400 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-b__llama31_4b__wikics__s42:244s;cuda2:trace_context:trace_bank__n1__Semantic-512-b__llama31_4b__wikics__s666:792s;cuda3:trace_context:trace_bank__n1__Full-256-b__llama3_8b__wikics__s42:372s | GPU cuda1:4885/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:97%260806-10:13:48 (UTC+8)
414tasks total=14270 running=3 done=862 failed=0 blocked=0 pending=13405 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s42:1804s;cuda2:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__wikics__s666:217s;cuda3:trace_context:trace_bank__n1__Full-128-b__llama3_8b__wikics__s42:777s | GPU cuda1:4885/24564MiB:96%;cuda2:4885/24564MiB:97%;cuda3:7851/24564MiB:98%260806-09:48:33 (UTC+8)
413tasks total=14270 running=3 done=858 failed=0 blocked=0 pending=13409 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-512-a__llama31_4b__wikics__s42:667s;cuda2:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__wikics__s666:156s;cuda3:trace_context:trace_bank__n1__Full-128-a__llama3_8b__wikics__s42:512s | GPU cuda1:4885/24564MiB:0%;cuda2:4565/24564MiB:94%;cuda3:7851/24564MiB:95%260806-09:29:35 (UTC+8)
412tasks total=14270 running=3 done=851 failed=0 blocked=0 pending=13416 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s42:1734s;cuda2:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s666:26s;cuda3:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s42:964s | GPU cuda1:4625/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:95%260806-09:01:07 (UTC+8)
411tasks total=14270 running=3 done=850 failed=0 blocked=0 pending=13417 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-b__llama31_4b__wikics__s42:785s;cuda2:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s666:648s;cuda3:trace_context:trace_bank__n1__Semantic-256-a__llama3_8b__wikics__s42:15s | GPU cuda1:4625/24564MiB:0%;cuda2:5445/24564MiB:100%;cuda3:7689/24564MiB:14%260806-08:45:22 (UTC+8)
410tasks total=14270 running=3 done=845 failed=0 blocked=0 pending=13422 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s42:2592s;cuda2:trace_context:trace_bank__n1__Random-512-b__llama31_4b__wikics__s666:618s;cuda3:trace_context:trace_bank__n1__Full-512-b__llama3_8b__wikics__s42:521s | GPU cuda1:4625/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:97%260806-08:29:35 (UTC+8)
409tasks total=14270 running=3 done=843 failed=0 blocked=0 pending=13424 | results done=394/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-256-a__llama31_4b__wikics__s42:1267s;cuda2:trace_context:trace_bank__n1__Random-512-a__llama31_4b__wikics__s666:213s;cuda3:trace_context:trace_bank__n1__Full-512-a__llama3_8b__wikics__s42:650s | GPU cuda1:4625/24564MiB:0%;cuda2:4885/24564MiB:95%;cuda3:7851/24564MiB:96%260806-08:07:24 (UTC+8)
408tasks total=14270 running=3 done=836 failed=0 blocked=0 pending=13431 | results done=393/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s42:2754s;cuda2:trace_context:trace_bank__n1__Random-256-a__llama31_4b__wikics__s666:186s;cuda3:g4_teacher:g4_teacher__llama3_8b__wikics__s42:72s | GPU cuda1:6067/24564MiB:26%;cuda2:4625/24564MiB:32%;cuda3:7851/24564MiB:96%260806-07:45:19 (UTC+8)
407tasks total=14270 running=3 done=768 failed=0 blocked=0 pending=13499 | results done=331/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-b__llama31_4b__wikics__s42:665s;cuda2:trace_context:trace_bank__n1__Full-1024-b__llama31_4b__wikics__s666:696s;cuda3:trace_downstream:run__T4_StudentOptGlobal__llama3_8b__wikics__s42__gcn3:78s | GPU cuda1:6067/24564MiB:26%;cuda2:5465/24564MiB:99%;cuda3:1637/24564MiB:84%260806-07:10:32 (UTC+8)
406tasks total=14270 running=3 done=754 failed=0 blocked=0 pending=13513 | results done=323/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s42:2238s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s666:1136s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s42:1460s | GPU cuda1:6067/24564MiB:25%;cuda2:5465/24564MiB:100%;cuda3:7851/24564MiB:0%260806-06:51:37 (UTC+8)
405tasks total=14270 running=3 done=754 failed=0 blocked=0 pending=13513 | results done=323/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s42:1667s;cuda2:trace_context:trace_bank__n1__Full-1024-a__llama31_4b__wikics__s666:565s;cuda3:trace_context:trace_bank__base__HopQuota-1024-b__llama3_8b__wikics__s42:889s | GPU cuda1:6067/24564MiB:25%;cuda2:5465/24564MiB:94%;cuda3:7851/24564MiB:96%260806-06:42:13 (UTC+8)
404tasks total=14270 running=3 done=750 failed=0 blocked=0 pending=13517 | results done=323/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s42:340s;cuda2:trace_context:trace_bank__n1__Full-256-a__llama31_4b__wikics__s666:540s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s42:1709s | GPU cuda1:6067/24564MiB:25%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:96%260806-06:20:00 (UTC+8)
403tasks total=14270 running=3 done=750 failed=0 blocked=0 pending=13517 | results done=323/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Diverse-128-a__llama31_4b__wikics__s42:157s;cuda2:trace_context:trace_bank__n1__Full-256-a__llama31_4b__wikics__s666:358s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s42:1526s | GPU cuda1:6067/24564MiB:0%;cuda2:4625/24564MiB:98%;cuda3:7851/24564MiB:96%260806-06:16:56 (UTC+8)
402tasks total=14270 running=3 done=746 failed=0 blocked=0 pending=13521 | results done=323/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-b__llama31_4b__wikics__s42:1621s;cuda2:trace_context:trace_bank__n1__Full-128-b__llama31_4b__wikics__s666:492s;cuda3:trace_context:trace_bank__base__HopQuota-1024-a__llama3_8b__wikics__s42:1140s | GPU cuda1:6947/24564MiB:99%;cuda2:4565/24564MiB:68%;cuda3:7851/24564MiB:100%260806-06:10:45 (UTC+8)
401tasks total=14270 running=3 done=738 failed=0 blocked=0 pending=13529 | results done=323/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s42:1773s;cuda2:trace_context:trace_bank__n1__Semantic-256-a__llama31_4b__wikics__s666:668s;cuda3:trace_context:trace_bank__base__HopQuota-512-b__llama3_8b__wikics__s42:774s | GPU cuda1:6947/24564MiB:100%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:97%260806-05:42:03 (UTC+8)
400tasks total=14270 running=3 done=734 failed=0 blocked=0 pending=13533 | results done=323/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-1024-a__llama31_4b__wikics__s42:67s;cuda2:trace_context:trace_bank__n1__Full-512-a__llama31_4b__wikics__s666:832s;cuda3:trace_context:trace_bank__base__HopQuota-512-a__llama3_8b__wikics__s42:404s | GPU cuda1:6947/24564MiB:100%;cuda2:4885/24564MiB:97%;cuda3:7851/24564MiB:97%260806-05:13:37 (UTC+8)
399tasks total=14270 running=3 done=711 failed=0 blocked=0 pending=13556 | results done=309/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-512-a__llama31_4b__wikics__s42:504s;cuda2:trace_downstream:run__C6_BudgetSweep__budget_512__llama31_4b__wikics__s666__gcn3:29s;cuda3:trace_context:trace_bank__base__HopQuota-256-a__llama3_8b__wikics__s42:531s | GPU cuda1:6387/24564MiB:100%;cuda2:1637/24564MiB:80%;cuda3:7851/24564MiB:98%260806-04:45:11 (UTC+8)
398tasks total=14270 running=3 done=659 failed=0 blocked=0 pending=13608 | results done=263/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-128-a__llama31_4b__wikics__s42:583s;cuda2:trace_downstream:run__T4_StudentOptGlobal__llama31_4b__wikics__s666__sage3:59s;cuda3:trace_context:trace_bank__base__HopQuota-128-b__llama3_8b__wikics__s42:183s | GPU cuda1:6067/24564MiB:98%;cuda2:1441/24564MiB:74%;cuda3:7851/24564MiB:95%260806-04:26:17 (UTC+8)
397tasks total=14270 running=3 done=638 failed=0 blocked=0 pending=13629 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s42:1069s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s666:860s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s42:4057s | GPU cuda1:6967/24564MiB:100%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:0%260806-04:04:08 (UTC+8)
396tasks total=14270 running=3 done=638 failed=0 blocked=0 pending=13629 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s42:694s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s666:485s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s42:3682s | GPU cuda1:6967/24564MiB:100%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:0%260806-03:57:54 (UTC+8)
395tasks total=14270 running=3 done=638 failed=0 blocked=0 pending=13629 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-b__llama31_4b__wikics__s42:514s;cuda2:trace_context:trace_bank__base__HopQuota-1024-b__llama31_4b__wikics__s666:304s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s42:3502s | GPU cuda1:6967/24564MiB:100%;cuda2:5445/24564MiB:17%;cuda3:7851/24564MiB:95%260806-03:54:50 (UTC+8)
394tasks total=14270 running=3 done=636 failed=0 blocked=0 pending=13631 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s42:1766s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s666:1208s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s42:2937s | GPU cuda1:6967/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260806-03:45:28 (UTC+8)
393tasks total=14270 running=3 done=636 failed=0 blocked=0 pending=13631 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-1024-a__llama31_4b__wikics__s42:628s;cuda2:trace_context:trace_bank__base__HopQuota-1024-a__llama31_4b__wikics__s666:70s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s42:1799s | GPU cuda1:6967/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260806-03:26:31 (UTC+8)
392tasks total=14270 running=3 done=630 failed=0 blocked=0 pending=13637 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-512-b__llama31_4b__wikics__s42:356s;cuda2:trace_context:trace_bank__base__HopQuota-512-a__llama31_4b__wikics__s666:509s;cuda3:trace_context:trace_bank__base__Diverse-1024-b__llama3_8b__wikics__s42:473s | GPU cuda1:6387/24564MiB:100%;cuda2:4905/24564MiB:97%;cuda3:7851/24564MiB:0%260806-03:04:23 (UTC+8)
391tasks total=14270 running=3 done=617 failed=0 blocked=0 pending=13650 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-256-a__llama31_4b__wikics__s42:602s;cuda2:trace_context:trace_bank__base__HopQuota-128-a__llama31_4b__wikics__s666:314s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s42:2609s | GPU cuda1:5907/24564MiB:98%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:0%260806-02:26:26 (UTC+8)
390tasks total=14270 running=3 done=611 failed=0 blocked=0 pending=13656 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Random-128-a__llama31_4b__wikics__s42:405s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s666:2648s;cuda3:trace_context:trace_bank__base__Diverse-1024-a__llama3_8b__wikics__s42:1284s | GPU cuda1:5995/24564MiB:97%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:0%260806-02:04:19 (UTC+8)
389tasks total=14270 running=3 done=606 failed=0 blocked=0 pending=13661 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-b__llama31_4b__wikics__s42:142s;cuda2:trace_context:trace_bank__base__Diverse-1024-b__llama31_4b__wikics__s666:561s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s42:2827s | GPU cuda1:6875/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260806-01:29:33 (UTC+8)
388tasks total=14270 running=3 done=604 failed=0 blocked=0 pending=13663 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s42:828s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s666:3102s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s42:1692s | GPU cuda1:6895/24564MiB:100%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:100%260806-01:10:37 (UTC+8)
387tasks total=14270 running=3 done=604 failed=0 blocked=0 pending=13663 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s42:452s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s666:2725s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s42:1316s | GPU cuda1:6895/24564MiB:95%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:100%260806-01:04:24 (UTC+8)
386tasks total=14270 running=3 done=604 failed=0 blocked=0 pending=13663 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s42:270s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s666:2544s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s42:1134s | GPU cuda1:6875/24564MiB:100%;cuda2:5445/24564MiB:0%;cuda3:7851/24564MiB:0%260806-01:01:18 (UTC+8)
385tasks total=14270 running=3 done=604 failed=0 blocked=0 pending=13663 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-1024-a__llama31_4b__wikics__s42:88s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s666:2362s;cuda3:trace_context:trace_bank__base__Diverse-512-b__llama3_8b__wikics__s42:952s | GPU cuda1:6875/24564MiB:95%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260806-00:58:17 (UTC+8)
384tasks total=14270 running=3 done=598 failed=0 blocked=0 pending=13669 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-128-b__llama31_4b__wikics__s42:442s;cuda2:trace_context:trace_bank__base__Diverse-1024-a__llama31_4b__wikics__s666:653s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s42:2802s | GPU cuda1:5995/24564MiB:68%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:0%260806-00:29:51 (UTC+8)
383tasks total=14270 running=3 done=592 failed=0 blocked=0 pending=13675 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s42:662s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s666:2595s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s42:1669s | GPU cuda1:6055/24564MiB:97%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:100%260806-00:10:53 (UTC+8)
382tasks total=14270 running=3 done=592 failed=0 blocked=0 pending=13675 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s42:284s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s666:2217s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s42:1291s | GPU cuda1:6055/24564MiB:98%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:97%260806-00:04:40 (UTC+8)
381tasks total=14270 running=3 done=592 failed=0 blocked=0 pending=13675 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-b__llama31_4b__wikics__s42:103s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s666:2036s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s42:1110s | GPU cuda1:6055/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260806-00:01:34 (UTC+8)
380tasks total=14270 running=3 done=591 failed=0 blocked=0 pending=13676 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Semantic-256-a__llama31_4b__wikics__s42:683s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s666:1853s;cuda3:trace_context:trace_bank__base__Diverse-512-a__llama3_8b__wikics__s42:927s | GPU cuda1:6055/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:0%260805-23:58:33 (UTC+8)
379tasks total=14270 running=3 done=586 failed=0 blocked=0 pending=13681 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-b__llama31_4b__wikics__s42:594s;cuda2:trace_context:trace_bank__base__Diverse-512-b__llama31_4b__wikics__s666:717s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s42:2946s | GPU cuda1:6335/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:0%260805-23:39:36 (UTC+8)
378tasks total=14270 running=3 done=584 failed=0 blocked=0 pending=13683 | results done=252/5050 | RUNNING cuda1:trace_context:trace_bank__n2__Full-512-a__llama31_4b__wikics__s42:890s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s666:3015s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s42:2187s | GPU cuda1:6335/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:0%260805-23:27:00 (UTC+8)
377tasks total=14270 running=3 done=575 failed=0 blocked=0 pending=13692 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-b__llama31_4b__wikics__s42:810s;cuda2:trace_context:trace_bank__base__Diverse-512-a__llama31_4b__wikics__s666:929s;cuda3:trace_context:trace_bank__base__Diverse-256-b__llama3_8b__wikics__s42:101s | GPU cuda1:6875/24564MiB:100%;cuda2:4885/24564MiB:0%;cuda3:7851/24564MiB:0%260805-22:52:12 (UTC+8)
376tasks total=14270 running=3 done=570 failed=0 blocked=0 pending=13697 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-1024-a__llama31_4b__wikics__s42:789s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s666:2012s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s42:1575s | GPU cuda1:6875/24564MiB:100%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:0%260805-22:23:45 (UTC+8)
375tasks total=14270 running=3 done=568 failed=0 blocked=0 pending=13699 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s42:661s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s666:876s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s42:439s | GPU cuda1:6315/24564MiB:100%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:0%260805-22:04:49 (UTC+8)
374tasks total=14270 running=3 done=568 failed=0 blocked=0 pending=13699 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s42:480s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s666:696s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s42:258s | GPU cuda1:6295/24564MiB:97%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:97%260805-22:01:46 (UTC+8)
373tasks total=14270 running=3 done=568 failed=0 blocked=0 pending=13699 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-512-b__llama31_4b__wikics__s42:298s;cuda2:trace_context:trace_bank__base__Diverse-256-b__llama31_4b__wikics__s666:514s;cuda3:trace_context:trace_bank__base__Diverse-256-a__llama3_8b__wikics__s42:76s | GPU cuda1:6295/24564MiB:97%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:0%260805-21:58:45 (UTC+8)
372tasks total=14270 running=3 done=561 failed=0 blocked=0 pending=13706 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-256-b__llama31_4b__wikics__s42:647s;cuda2:trace_context:trace_bank__base__Diverse-256-a__llama31_4b__wikics__s666:1970s;cuda3:trace_context:trace_bank__base__Diverse-128-b__llama3_8b__wikics__s42:1721s | GPU cuda1:6033/24564MiB:98%;cuda2:4625/24564MiB:0%;cuda3:7851/24564MiB:0%260805-21:36:40 (UTC+8)
371tasks total=14270 running=3 done=553 failed=0 blocked=0 pending=13714 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__HopQuota-128-a__llama31_4b__wikics__s42:274s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s666:2533s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s42:2623s | GPU cuda1:5993/24564MiB:98%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:0%260805-21:01:50 (UTC+8)
370tasks total=14270 running=3 done=551 failed=0 blocked=0 pending=13716 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s42:4162s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s666:2152s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s42:2242s | GPU cuda1:6873/24564MiB:29%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:95%260805-20:55:36 (UTC+8)
369tasks total=14270 running=3 done=551 failed=0 blocked=0 pending=13716 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s42:3214s;cuda2:trace_context:trace_bank__base__Diverse-128-b__llama31_4b__wikics__s666:1203s;cuda3:trace_context:trace_bank__base__Diverse-128-a__llama3_8b__wikics__s42:1293s | GPU cuda1:6873/24564MiB:23%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:0%260805-20:39:45 (UTC+8)
368tasks total=14270 running=3 done=547 failed=0 blocked=0 pending=13720 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s42:1696s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s666:2318s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s42:2120s | GPU cuda1:6961/24564MiB:96%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:96%260805-20:14:26 (UTC+8)
367tasks total=14270 running=3 done=547 failed=0 blocked=0 pending=13720 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-b__llama31_4b__wikics__s42:179s;cuda2:trace_context:trace_bank__base__Diverse-128-a__llama31_4b__wikics__s666:801s;cuda3:trace_context:trace_bank__base__Semantic-1024-b__llama3_8b__wikics__s42:603s | GPU cuda1:6961/24564MiB:29%;cuda2:4565/24564MiB:0%;cuda3:7851/24564MiB:96%260805-19:49:09 (UTC+8)
366tasks total=14270 running=3 done=542 failed=0 blocked=0 pending=13725 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s42:2969s;cuda2:trace_context:trace_bank__base__Semantic-1024-b__llama31_4b__wikics__s666:898s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s42:1425s | GPU cuda1:6961/24564MiB:26%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:96%260805-19:23:51 (UTC+8)
365tasks total=14270 running=3 done=541 failed=0 blocked=0 pending=13726 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s42:2024s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s666:1561s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s42:480s | GPU cuda1:6961/24564MiB:94%;cuda2:5445/24564MiB:93%;cuda3:7851/24564MiB:96%260805-19:08:04 (UTC+8)
364tasks total=14270 running=3 done=541 failed=0 blocked=0 pending=13726 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s42:1842s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s666:1378s;cuda3:trace_context:trace_bank__base__Semantic-1024-a__llama3_8b__wikics__s42:297s | GPU cuda1:6961/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:96%260805-19:05:01 (UTC+8)
363tasks total=14270 running=3 done=538 failed=0 blocked=0 pending=13729 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s42:897s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s666:434s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s42:833s | GPU cuda1:6961/24564MiB:27%;cuda2:5445/24564MiB:94%;cuda3:7851/24564MiB:97%260805-18:49:17 (UTC+8)
362tasks total=14270 running=3 done=538 failed=0 blocked=0 pending=13729 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s42:714s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s666:251s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s42:650s | GPU cuda1:6961/24564MiB:96%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260805-18:46:14 (UTC+8)
361tasks total=14270 running=3 done=538 failed=0 blocked=0 pending=13729 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-1024-a__llama31_4b__wikics__s42:531s;cuda2:trace_context:trace_bank__base__Semantic-1024-a__llama31_4b__wikics__s666:68s;cuda3:trace_context:trace_bank__base__Semantic-512-b__llama3_8b__wikics__s42:467s | GPU cuda1:6961/24564MiB:100%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:97%260805-18:43:12 (UTC+8)
360tasks total=14270 running=3 done=532 failed=0 blocked=0 pending=13735 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s42:3281s;cuda2:trace_context:trace_bank__base__Semantic-512-b__llama31_4b__wikics__s666:266s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s42:1189s | GPU cuda1:6401/24564MiB:97%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:97%260805-18:30:37 (UTC+8)
359tasks total=14270 running=3 done=531 failed=0 blocked=0 pending=13736 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-b__llama31_4b__wikics__s42:2329s;cuda2:trace_context:trace_bank__base__Semantic-512-a__llama31_4b__wikics__s666:259s;cuda3:trace_context:trace_bank__base__Semantic-512-a__llama3_8b__wikics__s42:238s | GPU cuda1:6401/24564MiB:96%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:97%260805-18:14:54 (UTC+8)
358tasks total=14270 running=3 done=516 failed=0 blocked=0 pending=13751 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s42:3024s;cuda2:trace_context:trace_bank__base__Random-1024-b__llama31_4b__wikics__s666:64s;cuda3:trace_context:trace_bank__base__Random-1024-b__llama3_8b__wikics__s42:1500s | GPU cuda1:6401/24564MiB:24%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:95%260805-17:27:18 (UTC+8)
357tasks total=14270 running=3 done=511 failed=0 blocked=0 pending=13756 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-512-a__llama31_4b__wikics__s42:553s;cuda2:trace_context:trace_bank__base__Random-512-b__llama31_4b__wikics__s666:93s;cuda3:trace_context:trace_bank__base__Random-1024-a__llama3_8b__wikics__s42:1342s | GPU cuda1:6333/24564MiB:100%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:96%260805-16:46:05 (UTC+8)
356tasks total=14270 running=3 done=501 failed=0 blocked=0 pending=13766 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s42:1847s;cuda2:trace_context:trace_bank__base__Random-256-a__llama31_4b__wikics__s666:410s;cuda3:trace_context:trace_bank__base__Random-512-b__llama3_8b__wikics__s42:897s | GPU cuda1:6053/24564MiB:0%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:97%260805-16:14:27 (UTC+8)
355tasks total=14270 running=3 done=493 failed=0 blocked=0 pending=13774 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-b__llama31_4b__wikics__s42:136s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s666:1313s;cuda3:trace_context:trace_bank__base__Random-512-a__llama3_8b__wikics__s42:637s | GPU cuda1:6053/24564MiB:32%;cuda2:5445/24564MiB:95%;cuda3:7851/24564MiB:97%260805-15:46:00 (UTC+8)
354tasks total=14270 running=3 done=489 failed=0 blocked=0 pending=13778 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s42:2357s;cuda2:trace_context:trace_bank__base__Full-1024-b__llama31_4b__wikics__s666:364s;cuda3:trace_context:trace_bank__base__Random-256-b__llama3_8b__wikics__s42:738s | GPU cuda1:5915/24564MiB:28%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:42%260805-15:30:12 (UTC+8)
353tasks total=14270 running=3 done=487 failed=0 blocked=0 pending=13780 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s42:1222s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s666:816s;cuda3:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s42:649s | GPU cuda1:6145/24564MiB:97%;cuda2:5445/24564MiB:100%;cuda3:7851/24564MiB:94%260805-15:11:14 (UTC+8)
352tasks total=14270 running=3 done=487 failed=0 blocked=0 pending=13780 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-256-a__llama31_4b__wikics__s42:1039s;cuda2:trace_context:trace_bank__base__Full-1024-a__llama31_4b__wikics__s666:634s;cuda3:trace_context:trace_bank__base__Random-256-a__llama3_8b__wikics__s42:467s | GPU cuda1:6145/24564MiB:31%;cuda2:5445/24564MiB:94%;cuda3:7851/24564MiB:98%260805-15:08:11 (UTC+8)
351tasks total=14270 running=3 done=479 failed=0 blocked=0 pending=13788 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s42:2931s;cuda2:trace_context:trace_bank__base__Full-256-b__llama31_4b__wikics__s666:156s;cuda3:trace_context:trace_bank__base__Random-128-b__llama3_8b__wikics__s42:206s | GPU cuda1:6105/24564MiB:28%;cuda2:4625/24564MiB:97%;cuda3:7851/24564MiB:96%260805-14:49:18 (UTC+8)
350tasks total=14270 running=3 done=470 failed=0 blocked=0 pending=13797 | results done=246/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-b__llama31_4b__wikics__s42:1225s;cuda2:trace_context:trace_bank__base__Full-128-a__llama31_4b__wikics__s666:127s;cuda3:trace_context:trace_bank__base__Full-1024-b__llama3_8b__wikics__s42:1689s | GPU cuda1:6105/24564MiB:27%;cuda2:4565/24564MiB:97%;cuda3:7851/24564MiB:96%260805-14:20:52 (UTC+8)
349tasks total=14270 running=3 done=451 failed=0 blocked=0 pending=13816 | results done=234/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s42:2533s;cuda2:fixed_downstream:run__T1_FixedFull512__llama31_4b__wikics__s666__gcn1:0s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s42:2295s | GPU cuda1:6105/24564MiB:27%;cuda2:14/24564MiB:0%;cuda3:7851/24564MiB:100%260805-13:52:25 (UTC+8)
348tasks total=14270 running=3 done=448 failed=0 blocked=0 pending=13819 | results done=234/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Diverse-128-a__llama31_4b__wikics__s42:825s;cuda2:trace_context:trace_bank__base__Full-512-a__llama31_4b__wikics__s666:144s;cuda3:trace_context:trace_bank__base__Full-1024-a__llama3_8b__wikics__s42:587s | GPU cuda1:6105/24564MiB:27%;cuda2:4885/24564MiB:96%;cuda3:7851/24564MiB:95%260805-13:23:57 (UTC+8)
347tasks total=14270 running=3 done=440 failed=0 blocked=0 pending=13827 | results done=234/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-b__llama31_4b__wikics__s42:981s;cuda2:sft:sft__trace__llama31_4b__wikics__s666:1574s;cuda3:trace_context:trace_bank__base__Full-256-a__llama3_8b__wikics__s42:979s | GPU cuda1:6781/24564MiB:100%;cuda2:7969/24564MiB:99%;cuda3:7851/24564MiB:94%260805-12:55:29 (UTC+8)
346tasks total=14270 running=3 done=426 failed=0 blocked=0 pending=13841 | results done=228/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s42:915s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:2271s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s42:831s | GPU cuda1:6783/24564MiB:100%;cuda2:7851/24564MiB:96%;cuda3:7851/24564MiB:97%260805-12:23:52 (UTC+8)
345tasks total=14270 running=3 done=426 failed=0 blocked=0 pending=13841 | results done=228/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-1024-a__llama31_4b__wikics__s42:347s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:1702s;cuda3:trace_context:trace_bank__base__Full-128-a__llama3_8b__wikics__s42:263s | GPU cuda1:6783/24564MiB:98%;cuda2:7851/24564MiB:0%;cuda3:7851/24564MiB:97%260805-12:14:30 (UTC+8)
344tasks total=14270 running=3 done=413 failed=0 blocked=0 pending=13854 | results done=222/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-512-a__llama31_4b__wikics__s42:988s;cuda2:lingu_bank:lingu_bank__llama3_8b__wikics__s42:184s;cuda3:trace_context:trace_bank__base__Semantic-256-a__llama3_8b__wikics__s42:1026s | GPU cuda1:6223/24564MiB:98%;cuda2:7851/24564MiB:100%;cuda3:7851/24564MiB:97%260805-11:49:09 (UTC+8)
343tasks total=14270 running=3 done=401 failed=0 blocked=0 pending=13866 | results done=216/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-b__llama31_4b__wikics__s42:80s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:5396s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s42:1085s | GPU cuda1:5903/24564MiB:22%;cuda2:11839/24564MiB:100%;cuda3:7851/24564MiB:97%260805-11:23:50 (UTC+8)
342tasks total=14270 running=3 done=400 failed=0 blocked=0 pending=13867 | results done=216/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Semantic-128-a__llama31_4b__wikics__s42:126s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:4832s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s42:520s | GPU cuda1:5903/24564MiB:98%;cuda2:11839/24564MiB:99%;cuda3:7851/24564MiB:96%260805-11:14:23 (UTC+8)
341tasks total=14270 running=3 done=398 failed=0 blocked=0 pending=13869 | results done=216/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s42:1767s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:4650s;cuda3:trace_context:trace_bank__base__Full-512-b__llama3_8b__wikics__s42:338s | GPU cuda1:6783/24564MiB:96%;cuda2:11839/24564MiB:100%;cuda3:7851/24564MiB:97%260805-11:11:19 (UTC+8)
340tasks total=14270 running=3 done=397 failed=0 blocked=0 pending=13870 | results done=216/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-b__llama31_4b__wikics__s42:824s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:3707s;cuda3:trace_context:trace_bank__base__Full-512-a__llama3_8b__wikics__s42:853s | GPU cuda1:6783/24564MiB:100%;cuda2:11839/24564MiB:100%;cuda3:7851/24564MiB:97%260805-10:55:35 (UTC+8)
339tasks total=14270 running=3 done=395 failed=0 blocked=0 pending=13872 | results done=216/5050 | RUNNING cuda1:trace_context:trace_bank__n1__Random-1024-a__llama31_4b__wikics__s42:1436s;cuda2:sft:sft__lingu__llama3_8b__wikics__s42:2760s;cuda3:trace_context:trace_bank__base__Self__llama3_8b__wikics__s42:628s | GPU cuda1:5445/24564MiB:100%;cuda2:11849/24564MiB:100%;cuda3:7851/24564MiB:98%260805-10:39:52 (UTC+8)
#07

615DRD-KD实验汇报三卡worker情况

▦ table
last update
260804-04:17:16
records
50
type
text
latest
run_total=2200 done=114 fail=2086 running=0 remaining=0 | | cuda1=14/24564MiB util=0% temp=37C | cuda2=14/24564MiB util=0% temp=43C | cuda3=14/24564MiB util=0% temp=37C
ID数值接收时间
50run_total=2200 done=114 fail=2086 running=0 remaining=0 | | cuda1=14/24564MiB util=0% temp=37C | cuda2=14/24564MiB util=0% temp=43C | cuda3=14/24564MiB util=0% temp=37C260804-04:17:16 (UTC+8)
49run_total=2200 done=114 fail=1552 running=0 remaining=534 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:15s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7691/24564MiB util=13% temp=46C | cuda3=14/24564MiB util=0% temp=37C260804-04:16:32 (UTC+8)
48run_total=2200 done=114 fail=1552 running=0 remaining=534 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:26369s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=50C | cuda3=14/24564MiB util=0% temp=38C260804-04:06:32 (UTC+8)
47run_total=2200 done=114 fail=1552 running=0 remaining=534 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:25768s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=51C | cuda3=14/24564MiB util=0% temp=40C260804-03:56:31 (UTC+8)
46run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:25166s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:26792s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=19% temp=56C | cuda3=7833/24564MiB util=21% temp=50C260804-03:46:30 (UTC+8)
45run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:24565s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:26192s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=56C | cuda3=7833/24564MiB util=20% temp=50C260804-03:36:29 (UTC+8)
44run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:23964s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:25590s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=56C | cuda3=7833/24564MiB util=20% temp=50C260804-03:26:28 (UTC+8)
43run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:23363s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:24990s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=21% temp=49C260804-03:16:26 (UTC+8)
42run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:22761s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:24387s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=20% temp=56C | cuda3=7833/24564MiB util=21% temp=49C260804-03:06:26 (UTC+8)
41run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:22160s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:23787s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=56C | cuda3=7833/24564MiB util=20% temp=50C260804-02:56:24 (UTC+8)
40run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:21560s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:23186s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=56C | cuda3=7833/24564MiB util=21% temp=49C260804-02:46:23 (UTC+8)
39run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:20958s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:22584s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=20% temp=55C | cuda3=7833/24564MiB util=21% temp=49C260804-02:36:22 (UTC+8)
38run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:20357s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:21983s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=21% temp=49C260804-02:26:21 (UTC+8)
37run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:19755s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:21381s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=36% temp=58C | cuda3=7833/24564MiB util=20% temp=50C260804-02:16:19 (UTC+8)
36run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:19152s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:20778s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=20% temp=56C | cuda3=7833/24564MiB util=19% temp=49C260804-02:06:18 (UTC+8)
35run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:17941s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:19567s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=20% temp=56C | cuda3=7833/24564MiB util=21% temp=48C260804-01:46:04 (UTC+8)
34run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:17340s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:18967s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=19% temp=49C260804-01:36:03 (UTC+8)
33run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:16739s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:18365s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=21% temp=49C260804-01:26:03 (UTC+8)
32run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:16139s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:17765s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=21% temp=49C260804-01:16:02 (UTC+8)
31run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:15538s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:17164s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=20% temp=55C | cuda3=7833/24564MiB util=18% temp=50C260804-01:06:01 (UTC+8)
30run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:14937s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:16564s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=20% temp=50C260804-00:56:00 (UTC+8)
29run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:14335s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:15961s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=56C | cuda3=7833/24564MiB util=19% temp=49C260804-00:46:00 (UTC+8)
28run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:13123s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:14750s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=56C | cuda3=7833/24564MiB util=20% temp=56C260804-00:25:47 (UTC+8)
27run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:12521s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:14147s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=21% temp=49C260804-00:15:46 (UTC+8)
26run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:11920s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:13546s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=21% temp=49C260804-00:05:43 (UTC+8)
25run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:11318s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:12945s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=20% temp=56C | cuda3=7833/24564MiB util=21% temp=49C260803-23:55:42 (UTC+8)
24run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:10717s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:12344s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=21% temp=56C | cuda3=7833/24564MiB util=21% temp=49C260803-23:45:41 (UTC+8)
23run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:10115s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:11742s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=19% temp=55C | cuda3=7833/24564MiB util=20% temp=49C260803-23:35:40 (UTC+8)
22run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:9514s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:11140s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=19% temp=56C | cuda3=7833/24564MiB util=21% temp=50C260803-23:25:38 (UTC+8)
21run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:8913s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:10539s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=19% temp=55C | cuda3=7833/24564MiB util=19% temp=49C260803-23:15:36 (UTC+8)
20run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:8312s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:9939s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=21% temp=48C260803-23:05:36 (UTC+8)
19run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:7712s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:9338s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=20% temp=56C | cuda3=7833/24564MiB util=19% temp=49C260803-22:55:35 (UTC+8)
18run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:7110s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:8737s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=21% temp=48C260803-22:45:34 (UTC+8)
17run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:6509s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:8135s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=20% temp=49C260803-22:35:33 (UTC+8)
16run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:5908s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:7534s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=21% temp=60C | cuda3=7833/24564MiB util=21% temp=48C260803-22:25:32 (UTC+8)
15run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:5305s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:6932s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=19% temp=55C | cuda3=7833/24564MiB util=21% temp=54C260803-22:15:30 (UTC+8)
14run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:4705s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:6331s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=20% temp=56C | cuda3=7833/24564MiB util=20% temp=49C260803-22:05:28 (UTC+8)
13run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:3492s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:5118s | cuda1=14/24564MiB util=0% temp=36C | cuda2=7833/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=21% temp=49C260803-21:45:15 (UTC+8)
12run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:2280s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:3906s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=20% temp=55C | cuda3=7833/24564MiB util=21% temp=50C260803-21:25:04 (UTC+8)
11run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:1678s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:3305s | cuda1=14/24564MiB util=0% temp=37C | cuda2=7833/24564MiB util=21% temp=56C | cuda3=7833/24564MiB util=19% temp=49C260803-21:15:03 (UTC+8)
10run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:1077s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:2704s | cuda1=14/24564MiB util=0% temp=38C | cuda2=7829/24564MiB util=21% temp=55C | cuda3=7833/24564MiB util=19% temp=49C260803-21:05:01 (UTC+8)
9run_total=2200 done=114 fail=1012 running=0 remaining=1074 | duo_teacher:duo_teacher__duo__llama3_8b__cora__s666:476s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:2102s | cuda1=14/24564MiB util=0% temp=39C | cuda2=7827/24564MiB util=20% temp=56C | cuda3=7833/24564MiB util=20% temp=49C260803-20:55:00 (UTC+8)
8run_total=2200 done=78 fail=1012 running=1 remaining=1110 | downstream:run__CalibratedJS_40__drd__llama3_8b__cora__s666__gcn3:5s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:1501s | cuda1=14/24564MiB util=0% temp=38C | cuda2=671/24564MiB util=20% temp=55C | cuda3=7833/24564MiB util=20% temp=49C260803-20:44:58 (UTC+8)
7run_total=2200 done=76 fail=1012 running=0 remaining=1112 | teacher:teacher_a__drd__llama3_8b__cora__s666:408s | duo_teacher:duo_teacher__duo__llama3_8b__cora__s42:898s | cuda1=14/24564MiB util=0% temp=39C | cuda2=7851/24564MiB util=100% temp=74C | cuda3=7829/24564MiB util=19% temp=49C260803-20:34:57 (UTC+8)
6run_total=2200 done=38 fail=1012 running=0 remaining=1150 | teacher:teacher_b__drd__llama3_8b__cora__s42:93s | sft:sft__drd__llama3_8b__cora__s666:348s | cuda1=7851/24564MiB util=72% temp=59C | cuda2=7557/24564MiB util=100% temp=75C | cuda3=14/24564MiB util=0% temp=46C260803-20:14:44 (UTC+8)
5run_total=2200 done=0 fail=0 running=0 remaining=2200 | teacher:teacher_a__drd__llama3_8b__cora__s42:212s | teacher:teacher_b__drd__llama31_4b__cora__s42:143s | teacher:teacher_b__lingu__llama3_8b__cora__s42:497s | cuda1=7851/24564MiB util=92% temp=67C | cuda2=4565/24564MiB util=44% temp=72C | cuda3=7851/24564MiB util=89% temp=78C260803-20:04:42 (UTC+8)
4run_total=2200 done=0 fail=0 running=0 remaining=2200 | sft:sft__drd__llama3_8b__cora__s42:760s | teacher:teacher_a__drd__llama31_4b__cora__s42:19s | teacher:teacher_a__lingu__llama3_8b__cora__s42:227s | cuda1=7577/24564MiB util=100% temp=67C | cuda2=4685/24564MiB util=93% temp=77C | cuda3=7851/24564MiB util=96% temp=77C260803-19:54:41 (UTC+8)
3run_total=2200 done=0 fail=0 running=0 remaining=2200 | sft:sft__drd__llama3_8b__cora__s42:160s | sft:sft__drd__llama31_4b__cora__s42:163s | sft:sft__lingu__llama3_8b__cora__s42:157s | cuda1=7575/24564MiB util=99% temp=65C | cuda2=5389/24564MiB util=72% temp=78C | cuda3=7017/24564MiB util=100% temp=74C260803-19:44:39 (UTC+8)
2run_total=2200 done=0 fail=0 running=0 remaining=2200 | task_only:taskonly__arxiv__s888__sage3:45s | task_only:taskonly__arxiv__s888__gcn3:99s | task_only:taskonly__arxiv__s999__gcn1:3s | cuda1=7839/24564MiB util=96% temp=65C | cuda2=8753/24564MiB util=94% temp=78C | cuda3=401/24564MiB util=0% temp=63C260803-19:34:39 (UTC+8)
1run_total=2200 done=0 fail=0 running=0 remaining=2200 | | cuda1=14/24564MiB util=0% temp=38C | cuda2=14/24564MiB util=0% temp=40C | cuda3=14/24564MiB util=0% temp=40C260803-19:24:37 (UTC+8)
#08

615Graft实验汇报三卡worker情况

▦ table
last update
260802-15:52:41
records
123
type
text
latest
run_total=360 done=67 fail=120 running=2 remaining=173 | cuda1=401/24564MiB util=0% temp=49C | cuda2=7381/24564MiB util=82% temp=73C | cuda3=1273/24564MiB util=80% temp=64C
ID数值接收时间
123run_total=360 done=67 fail=120 running=2 remaining=173 | cuda1=401/24564MiB util=0% temp=49C | cuda2=7381/24564MiB util=82% temp=73C | cuda3=1273/24564MiB util=80% temp=64C260802-15:52:41 (UTC+8)
122run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=80% temp=63C | cuda2=7381/24564MiB util=77% temp=78C | cuda3=7381/24564MiB util=80% temp=73C260802-15:32:29 (UTC+8)
121run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=78% temp=63C | cuda2=401/24564MiB util=0% temp=67C | cuda3=7381/24564MiB util=82% temp=74C260802-15:22:28 (UTC+8)
120run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=75% temp=62C | cuda2=7381/24564MiB util=82% temp=80C | cuda3=7381/24564MiB util=81% temp=73C260802-15:12:26 (UTC+8)
119run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=80% temp=63C | cuda2=7381/24564MiB util=80% temp=79C | cuda3=7381/24564MiB util=79% temp=74C260802-15:02:25 (UTC+8)
118run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=79% temp=63C | cuda2=7381/24564MiB util=81% temp=79C | cuda3=7381/24564MiB util=81% temp=74C260802-14:52:24 (UTC+8)
117run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=79% temp=63C | cuda2=7381/24564MiB util=82% temp=79C | cuda3=7381/24564MiB util=81% temp=73C260802-14:42:23 (UTC+8)
116run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=81% temp=62C | cuda2=4565/24564MiB util=44% temp=66C | cuda3=4565/24564MiB util=45% temp=64C260802-14:32:21 (UTC+8)
115run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4565/24564MiB util=40% temp=55C | cuda2=4565/24564MiB util=46% temp=66C | cuda3=4565/24564MiB util=44% temp=63C260802-14:22:20 (UTC+8)
114run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4565/24564MiB util=45% temp=55C | cuda2=4565/24564MiB util=49% temp=67C | cuda3=4565/24564MiB util=44% temp=64C260802-14:12:19 (UTC+8)
113run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4565/24564MiB util=47% temp=56C | cuda2=4565/24564MiB util=46% temp=67C | cuda3=4565/24564MiB util=45% temp=63C260802-14:02:16 (UTC+8)
112run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4565/24564MiB util=47% temp=56C | cuda2=4565/24564MiB util=48% temp=67C | cuda3=4565/24564MiB util=47% temp=64C260802-13:52:14 (UTC+8)
111run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4565/24564MiB util=48% temp=56C | cuda2=4565/24564MiB util=44% temp=68C | cuda3=4565/24564MiB util=45% temp=64C260802-13:42:11 (UTC+8)
110run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=67% temp=66C | cuda2=4565/24564MiB util=48% temp=68C | cuda3=4565/24564MiB util=47% temp=66C260802-13:32:10 (UTC+8)
109run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=62% temp=66C | cuda2=4565/24564MiB util=47% temp=69C | cuda3=4565/24564MiB util=46% temp=66C260802-13:22:09 (UTC+8)
108run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=100% temp=66C | cuda2=4565/24564MiB util=48% temp=69C | cuda3=4565/24564MiB util=47% temp=65C260802-13:12:08 (UTC+8)
107run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=100% temp=63C | cuda2=4565/24564MiB util=40% temp=67C | cuda3=4565/24564MiB util=45% temp=66C260802-13:02:07 (UTC+8)
106run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=100% temp=66C | cuda2=7381/24564MiB util=74% temp=80C | cuda3=4549/24564MiB util=2% temp=56C260802-12:52:06 (UTC+8)
105run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=75% temp=67C | cuda2=7381/24564MiB util=59% temp=83C | cuda3=7381/24564MiB util=78% temp=69C260802-12:42:04 (UTC+8)
104run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=73% temp=67C | cuda2=7381/24564MiB util=74% temp=84C | cuda3=7381/24564MiB util=100% temp=77C260802-12:32:02 (UTC+8)
103run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=73% temp=67C | cuda2=7381/24564MiB util=73% temp=84C | cuda3=7381/24564MiB util=52% temp=78C260802-12:22:00 (UTC+8)
102run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=76% temp=65C | cuda2=7381/24564MiB util=79% temp=84C | cuda3=7381/24564MiB util=59% temp=77C260802-12:11:58 (UTC+8)
101run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=89% temp=68C | cuda2=7381/24564MiB util=76% temp=84C | cuda3=7381/24564MiB util=78% temp=77C260802-12:01:56 (UTC+8)
100run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=60% temp=67C | cuda2=7381/24564MiB util=74% temp=84C | cuda3=7381/24564MiB util=100% temp=74C260802-11:51:53 (UTC+8)
99run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=99% temp=63C | cuda2=7381/24564MiB util=95% temp=83C | cuda3=7381/24564MiB util=73% temp=78C260802-11:41:51 (UTC+8)
98run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=63% temp=67C | cuda2=7381/24564MiB util=87% temp=85C | cuda3=7381/24564MiB util=87% temp=78C260802-11:31:48 (UTC+8)
97run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=100% temp=66C | cuda2=7381/24564MiB util=85% temp=85C | cuda3=7381/24564MiB util=85% temp=78C260802-11:21:47 (UTC+8)
96run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=84% temp=67C | cuda2=7381/24564MiB util=88% temp=84C | cuda3=7381/24564MiB util=100% temp=78C260802-11:11:45 (UTC+8)
95run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=93% temp=66C | cuda2=7381/24564MiB util=96% temp=85C | cuda3=7381/24564MiB util=100% temp=78C260802-11:01:45 (UTC+8)
94run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=100% temp=67C | cuda2=7381/24564MiB util=84% temp=85C | cuda3=7381/24564MiB util=87% temp=79C260802-10:51:44 (UTC+8)
93run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=80% temp=67C | cuda2=7381/24564MiB util=87% temp=84C | cuda3=7381/24564MiB util=98% temp=78C260802-10:41:42 (UTC+8)
92run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=80% temp=67C | cuda2=7381/24564MiB util=93% temp=85C | cuda3=7381/24564MiB util=85% temp=79C260802-10:31:41 (UTC+8)
91run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=100% temp=65C | cuda2=7381/24564MiB util=87% temp=86C | cuda3=7381/24564MiB util=92% temp=78C260802-10:21:38 (UTC+8)
90run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=59% temp=64C | cuda2=7381/24564MiB util=87% temp=85C | cuda3=7381/24564MiB util=100% temp=73C260802-10:11:36 (UTC+8)
89run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=100% temp=67C | cuda2=7381/24564MiB util=100% temp=86C | cuda3=7381/24564MiB util=87% temp=78C260802-10:01:36 (UTC+8)
88run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=86% temp=67C | cuda2=7381/24564MiB util=88% temp=84C | cuda3=7381/24564MiB util=100% temp=78C260802-09:51:35 (UTC+8)
87run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=95% temp=66C | cuda2=7381/24564MiB util=99% temp=82C | cuda3=7381/24564MiB util=100% temp=77C260802-09:41:33 (UTC+8)
86run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=85% temp=67C | cuda2=7381/24564MiB util=87% temp=84C | cuda3=7381/24564MiB util=86% temp=78C260802-09:31:33 (UTC+8)
85run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=88% temp=67C | cuda2=7381/24564MiB util=100% temp=84C | cuda3=7381/24564MiB util=85% temp=78C260802-09:21:31 (UTC+8)
84run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=88% temp=66C | cuda2=7381/24564MiB util=95% temp=81C | cuda3=7381/24564MiB util=84% temp=76C260802-09:11:29 (UTC+8)
83run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=87% temp=67C | cuda2=7381/24564MiB util=89% temp=84C | cuda3=7381/24564MiB util=87% temp=78C260802-09:01:29 (UTC+8)
82run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=100% temp=66C | cuda2=7381/24564MiB util=100% temp=83C | cuda3=7381/24564MiB util=89% temp=78C260802-08:41:17 (UTC+8)
81run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=89% temp=68C | cuda2=7381/24564MiB util=100% temp=83C | cuda3=7381/24564MiB util=88% temp=78C260802-08:31:16 (UTC+8)
80run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=89% temp=68C | cuda2=7381/24564MiB util=87% temp=85C | cuda3=7381/24564MiB util=85% temp=76C260802-08:21:14 (UTC+8)
79run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7381/24564MiB util=91% temp=64C | cuda2=4725/24564MiB util=75% temp=80C | cuda3=7381/24564MiB util=88% temp=78C260802-08:11:14 (UTC+8)
78run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=78% temp=59C | cuda2=4725/24564MiB util=75% temp=82C | cuda3=7381/24564MiB util=90% temp=77C260802-07:40:53 (UTC+8)
77run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=90% temp=65C | cuda2=4725/24564MiB util=86% temp=73C | cuda3=7381/24564MiB util=100% temp=74C260802-07:30:48 (UTC+8)
76run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=80% temp=65C | cuda2=4725/24564MiB util=69% temp=81C | cuda3=4725/24564MiB util=67% temp=68C260802-07:20:46 (UTC+8)
75run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=88% temp=62C | cuda2=4725/24564MiB util=70% temp=83C | cuda3=4725/24564MiB util=95% temp=75C260802-07:10:44 (UTC+8)
74run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=49% temp=59C | cuda2=4725/24564MiB util=74% temp=77C | cuda3=4725/24564MiB util=79% temp=75C260802-07:00:44 (UTC+8)
73run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=100% temp=58C | cuda2=4725/24564MiB util=100% temp=81C | cuda3=4725/24564MiB util=83% temp=76C260802-06:50:42 (UTC+8)
72run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=86% temp=66C | cuda2=4725/24564MiB util=88% temp=82C | cuda3=4725/24564MiB util=90% temp=71C260802-06:30:31 (UTC+8)
71run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=87% temp=67C | cuda2=4725/24564MiB util=90% temp=83C | cuda3=4725/24564MiB util=88% temp=76C260802-06:20:30 (UTC+8)
70run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=90% temp=66C | cuda2=4725/24564MiB util=89% temp=81C | cuda3=4725/24564MiB util=86% temp=74C260802-06:10:29 (UTC+8)
69run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=90% temp=65C | cuda2=4725/24564MiB util=90% temp=83C | cuda3=4725/24564MiB util=89% temp=76C260802-06:00:28 (UTC+8)
68run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=89% temp=66C | cuda2=4725/24564MiB util=88% temp=82C | cuda3=4725/24564MiB util=90% temp=76C260802-05:50:27 (UTC+8)
67run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=89% temp=66C | cuda2=4725/24564MiB util=100% temp=81C | cuda3=4725/24564MiB util=91% temp=77C260802-05:40:26 (UTC+8)
66run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=4725/24564MiB util=89% temp=66C | cuda2=4725/24564MiB util=87% temp=83C | cuda3=4725/24564MiB util=87% temp=76C260802-05:30:24 (UTC+8)
65run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=99% temp=66C | cuda2=4725/24564MiB util=98% temp=79C | cuda3=4725/24564MiB util=89% temp=76C260802-05:20:22 (UTC+8)
64run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=67C | cuda2=4725/24564MiB util=89% temp=82C | cuda3=4725/24564MiB util=91% temp=77C260802-05:10:21 (UTC+8)
63run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=66C | cuda2=4725/24564MiB util=93% temp=82C | cuda3=4725/24564MiB util=90% temp=75C260802-05:00:19 (UTC+8)
62run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=65C | cuda2=4725/24564MiB util=89% temp=79C | cuda3=4725/24564MiB util=88% temp=74C260802-04:50:16 (UTC+8)
61run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=99% temp=66C | cuda2=4725/24564MiB util=89% temp=83C | cuda3=4725/24564MiB util=87% temp=76C260802-04:40:14 (UTC+8)
60run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=66C | cuda2=4725/24564MiB util=90% temp=82C | cuda3=4725/24564MiB util=87% temp=76C260802-04:30:11 (UTC+8)
59run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=99% temp=67C | cuda2=4725/24564MiB util=88% temp=84C | cuda3=4705/24564MiB util=87% temp=77C260802-04:20:11 (UTC+8)
58run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=66C | cuda2=4705/24564MiB util=92% temp=80C | cuda3=4705/24564MiB util=91% temp=77C260802-04:10:10 (UTC+8)
57run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=68C | cuda2=4685/24564MiB util=91% temp=81C | cuda3=4685/24564MiB util=83% temp=75C260802-04:00:09 (UTC+8)
56run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=401/24564MiB util=0% temp=41C | cuda2=7527/24564MiB util=100% temp=79C | cuda3=7509/24564MiB util=100% temp=70C260802-03:50:08 (UTC+8)
55run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=100% temp=67C | cuda2=7527/24564MiB util=100% temp=84C | cuda3=7509/24564MiB util=99% temp=77C260802-03:40:07 (UTC+8)
54run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=99% temp=66C | cuda2=7527/24564MiB util=100% temp=86C | cuda3=7509/24564MiB util=100% temp=79C260802-03:30:06 (UTC+8)
53run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=100% temp=67C | cuda2=7527/24564MiB util=100% temp=84C | cuda3=7509/24564MiB util=100% temp=78C260802-03:20:06 (UTC+8)
52run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=99% temp=67C | cuda2=7527/24564MiB util=83% temp=86C | cuda3=7507/24564MiB util=100% temp=78C260802-03:10:04 (UTC+8)
51run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=100% temp=67C | cuda2=7527/24564MiB util=100% temp=85C | cuda3=7507/24564MiB util=100% temp=78C260802-03:00:02 (UTC+8)
50run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=100% temp=67C | cuda2=7527/24564MiB util=100% temp=85C | cuda3=7507/24564MiB util=99% temp=79C260802-02:49:56 (UTC+8)
49run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=100% temp=66C | cuda2=7527/24564MiB util=100% temp=85C | cuda3=7507/24564MiB util=99% temp=78C260802-02:39:53 (UTC+8)
48run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=100% temp=66C | cuda2=7251/24564MiB util=15% temp=48C | cuda3=401/24564MiB util=0% temp=47C260802-02:29:52 (UTC+8)
47run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7487/24564MiB util=92% temp=65C | cuda2=7547/24564MiB util=100% temp=78C | cuda3=7527/24564MiB util=100% temp=71C260802-02:19:49 (UTC+8)
46run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=401/24564MiB util=0% temp=42C | cuda2=7547/24564MiB util=100% temp=85C | cuda3=7527/24564MiB util=100% temp=77C260802-02:09:46 (UTC+8)
45run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=67C | cuda2=7547/24564MiB util=100% temp=85C | cuda3=7527/24564MiB util=99% temp=79C260802-01:49:32 (UTC+8)
44run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=99% temp=68C | cuda2=7547/24564MiB util=100% temp=85C | cuda3=7527/24564MiB util=100% temp=78C260802-01:39:29 (UTC+8)
43run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=66C | cuda2=7547/24564MiB util=99% temp=86C | cuda3=7527/24564MiB util=100% temp=79C260802-01:29:27 (UTC+8)
42run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=67C | cuda2=7547/24564MiB util=99% temp=85C | cuda3=7527/24564MiB util=100% temp=78C260802-01:19:25 (UTC+8)
41run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=67C | cuda2=7547/24564MiB util=100% temp=85C | cuda3=7527/24564MiB util=100% temp=78C260802-01:09:24 (UTC+8)
40run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=99% temp=67C | cuda2=7547/24564MiB util=99% temp=84C | cuda3=7527/24564MiB util=100% temp=77C260802-00:59:23 (UTC+8)
39run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=99% temp=65C | cuda2=401/24564MiB util=0% temp=57C | cuda3=7527/24564MiB util=100% temp=77C260802-00:49:22 (UTC+8)
38run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=99% temp=67C | cuda2=7507/24564MiB util=100% temp=83C | cuda3=401/24564MiB util=0% temp=54C260802-00:39:21 (UTC+8)
37run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=100% temp=63C | cuda2=7507/24564MiB util=100% temp=79C | cuda3=7507/24564MiB util=99% temp=71C260802-00:29:21 (UTC+8)
36run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=67C | cuda2=7507/24564MiB util=100% temp=84C | cuda3=7507/24564MiB util=99% temp=79C260802-00:09:09 (UTC+8)
35run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=67C | cuda2=7507/24564MiB util=100% temp=85C | cuda3=7507/24564MiB util=100% temp=78C260801-23:59:06 (UTC+8)
34run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=99% temp=67C | cuda2=7507/24564MiB util=100% temp=84C | cuda3=7507/24564MiB util=100% temp=77C260801-23:49:05 (UTC+8)
33run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=67C | cuda2=7507/24564MiB util=99% temp=86C | cuda3=7507/24564MiB util=100% temp=77C260801-23:39:05 (UTC+8)
32run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=67C | cuda2=7507/24564MiB util=100% temp=85C | cuda3=7507/24564MiB util=100% temp=78C260801-23:29:04 (UTC+8)
31run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=99% temp=67C | cuda2=7507/24564MiB util=100% temp=85C | cuda3=7507/24564MiB util=100% temp=78C260801-23:19:03 (UTC+8)
30run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=100% temp=66C | cuda2=7507/24564MiB util=100% temp=85C | cuda3=7507/24564MiB util=100% temp=78C260801-23:09:01 (UTC+8)
29run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=95% temp=67C | cuda2=401/24564MiB util=0% temp=49C | cuda3=401/24564MiB util=0% temp=47C260801-22:59:00 (UTC+8)
28run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7507/24564MiB util=99% temp=67C | cuda2=5389/24564MiB util=100% temp=84C | cuda3=7469/24564MiB util=99% temp=78C260801-22:48:59 (UTC+8)
27run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=401/24564MiB util=0% temp=45C | cuda2=5389/24564MiB util=93% temp=83C | cuda3=7469/24564MiB util=99% temp=78C260801-22:38:56 (UTC+8)
26run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7509/24564MiB util=99% temp=68C | cuda2=5389/24564MiB util=100% temp=85C | cuda3=7507/24564MiB util=100% temp=79C260801-22:28:53 (UTC+8)
25run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=66C | cuda2=5389/24564MiB util=100% temp=81C | cuda3=7487/24564MiB util=99% temp=73C260801-22:18:53 (UTC+8)
24run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=7527/24564MiB util=100% temp=67C | cuda2=5389/24564MiB util=99% temp=85C | cuda3=7507/24564MiB util=100% temp=78C260801-22:08:51 (UTC+8)
23run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5391/24564MiB util=96% temp=65C | cuda2=401/24564MiB util=0% temp=56C | cuda3=401/24564MiB util=0% temp=58C260801-21:58:50 (UTC+8)
22run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5391/24564MiB util=99% temp=66C | cuda2=5391/24564MiB util=100% temp=84C | cuda3=5411/24564MiB util=100% temp=78C260801-21:48:48 (UTC+8)
21run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5391/24564MiB util=96% temp=65C | cuda2=5391/24564MiB util=100% temp=84C | cuda3=5409/24564MiB util=100% temp=77C260801-21:38:47 (UTC+8)
20run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5391/24564MiB util=99% temp=66C | cuda2=5391/24564MiB util=100% temp=84C | cuda3=5409/24564MiB util=100% temp=77C260801-21:28:42 (UTC+8)
19run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=99% temp=67C | cuda2=5389/24564MiB util=100% temp=84C | cuda3=5409/24564MiB util=100% temp=76C260801-21:18:41 (UTC+8)
18run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=100% temp=65C | cuda2=5389/24564MiB util=100% temp=80C | cuda3=5409/24564MiB util=99% temp=74C260801-21:08:40 (UTC+8)
17run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=401/24564MiB util=0% temp=41C | cuda2=5389/24564MiB util=99% temp=77C | cuda3=401/24564MiB util=0% temp=46C260801-20:58:37 (UTC+8)
16run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=99% temp=67C | cuda2=401/24564MiB util=0% temp=61C | cuda3=5409/24564MiB util=95% temp=77C260801-20:48:35 (UTC+8)
15run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=100% temp=66C | cuda2=5409/24564MiB util=100% temp=84C | cuda3=5409/24564MiB util=100% temp=77C260801-20:38:35 (UTC+8)
14run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=100% temp=66C | cuda2=5409/24564MiB util=99% temp=85C | cuda3=5409/24564MiB util=100% temp=77C260801-20:28:34 (UTC+8)
13run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=99% temp=67C | cuda2=5389/24564MiB util=100% temp=83C | cuda3=5409/24564MiB util=99% temp=76C260801-20:08:23 (UTC+8)
12run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=401/24564MiB util=0% temp=41C | cuda2=5389/24564MiB util=100% temp=77C | cuda3=5409/24564MiB util=99% temp=71C260801-19:58:21 (UTC+8)
11run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=100% temp=66C | cuda2=5387/24564MiB util=100% temp=84C | cuda3=5409/24564MiB util=99% temp=77C260801-19:48:20 (UTC+8)
10run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=100% temp=67C | cuda2=5409/24564MiB util=99% temp=84C | cuda3=5409/24564MiB util=99% temp=77C260801-19:38:19 (UTC+8)
9run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=100% temp=65C | cuda2=5409/24564MiB util=100% temp=84C | cuda3=5409/24564MiB util=100% temp=77C260801-19:28:17 (UTC+8)
8run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=100% temp=65C | cuda2=5409/24564MiB util=100% temp=84C | cuda3=5409/24564MiB util=100% temp=77C260801-19:07:57 (UTC+8)
7run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5389/24564MiB util=100% temp=67C | cuda2=5409/24564MiB util=99% temp=85C | cuda3=5409/24564MiB util=100% temp=77C260801-18:57:56 (UTC+8)
6run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=401/24564MiB util=0% temp=42C | cuda2=5409/24564MiB util=100% temp=73C | cuda3=401/24564MiB util=0% temp=44C260801-18:47:56 (UTC+8)
5run_total=360 done=0 fail=120 running=0 remaining=240 | cuda1=5369/24564MiB util=100% temp=64C | cuda2=3925/24564MiB util=0% temp=57C | cuda3=5387/24564MiB util=78% temp=77C260801-18:37:55 (UTC+8)
4run_total=360 done=0 fail=0 running=0 remaining=360 | cuda1=14/24564MiB util=0% temp=38C | cuda2=14/24564MiB util=0% temp=40C | cuda3=14/24564MiB util=0% temp=41C260801-18:17:43 (UTC+8)
3run_total=360 done=0 fail=0 running=0 remaining=360 | cuda1=14/24564MiB util=0% temp=45C | cuda2=14/24564MiB util=0% temp=58C | cuda3=14/24564MiB util=0% temp=54C260801-16:49:58 (UTC+8)
2run_total=360 done=0 fail=0 running=0 remaining=360 | cuda1=14/24564MiB util=0% temp=45C | cuda2=14/24564MiB util=0% temp=58C | cuda3=14/24564MiB util=0% temp=55C260801-16:41:49 (UTC+8)
1run_total=360 done=0 fail=0 running=0 remaining=360 | cuda1=14/24564MiB util=0% temp=37C | cuda2=14/24564MiB util=0% temp=39C | cuda3=14/24564MiB util=0% temp=40C260801-16:32:46 (UTC+8)
λ LAB MONITOR · powered by Flask + Chart.js · built for science