WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so
bin /nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so...
No neptune configured, set NEPTUNE_API_TOKEN env var.

    Example runs on 4 GPUs:
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='decapoda-research/llama-7b-hf' --data_path=data/config.json --run_id=0 &> 0.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='decapoda-research/llama-30b-hf' --data_path=data/config.json --batch_size=16 --micro_batch_size=1 --run_id=1 --save_code=True &> 1.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='EleutherAI/gpt-j-6B' --data_path=data/config.json --run_id=2 &> 2.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='EleutherAI/gpt-neox-20b' --data_path=data/config.json --run_id=8 --batch_size=16 --micro_batch_size=4 &> 8.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='togethercomputer/GPT-NeoXT-Chat-Base-20B' --data_path=data/config.json --prompt_type='dai_faq' --run_id=13 --batch_size=16 --micro_batch_size=4 --num_epochs=100 --val_set_size=0 data_mix_in_path='' &> 13.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='togethercomputer/GPT-NeoXT-Chat-Base-20B' --data_path=data/config.json --run_id=28 --batch_size=16 --micro_batch_size=4 --num_epochs=8 --val_set_size=0 --data_mix_in_factor=0.1 --data_mix_in_prompt_type='human_bot' --save_code=True --cutoff_len=512  &> 28.log

    All metrics:
    CUDA_VISIBLE_DEVICES= finetune.py --data_mix_in_factor=0 --eval_steps=100 --warmup_steps=2 --val_set_size=100 --val_metrics="['bleu', 'rouge', 'sacrebleu', 'meteor']"

    # Fine-tune 20B on 24GB GPUs across 3 nodes with 3+2+2 GPUs
    rippa>
NCCL_P2P_LEVEL=LOC WORLD_SIZE=7 CUDA_VISIBLE_DEVICES="0,1,2" torchrun --node_rank 0 --nproc_per_node=3 --master_port=1234 --nnodes=3 --master_addr=10.10.10.2 finetune.py --data_path=merged_shuffled_OIG_87f6a1e788.json --micro_batch_size=1 --batch_size=7 --cutoff_len=512 --run_id=17 &>log.17.rank0
    ova>
NCCL_P2P_LEVEL=LOC WORLD_SIZE=7 CUDA_VISIBLE_DEVICES="0,1" torchrun --node_rank 1 --nproc_per_node=2 --master_port=1234 --nnodes=3 --master_addr=10.10.10.2 finetune.py --data_path=merged_shuffled_OIG_87f6a1e788.json --micro_batch_size=1 --batch_size=7 --cutoff_len=512 --run_id=17 &>log.17.rank1
    timemachine>
NCCL_P2P_LEVEL=LOC WORLD_SIZE=7 CUDA_VISIBLE_DEVICES="0,1" torchrun --node_rank 2 --nproc_per_node=2 --master_port=1234 --nnodes=3 --master_addr=10.10.10.2 finetune.py --data_path=merged_shuffled_OIG_87f6a1e788.json --micro_batch_size=1 --batch_size=7 --cutoff_len=512 --run_id=17 &>log.17.rank2

    
local_rank: 0
global rank: 0
local_rank: 1
global rank: 1
Training model with params:
save_code: False
run_id: 14
base_model: EleutherAI/pythia-6.9b
tokenizer_base_model: EleutherAI/pythia-6.9b
data_path: openassistant_oasst1.json
data_col_dict: None
valid_path: None
data_mix_in_path: 0-hero/OIG-small-chip2
data_mix_in_factor: 0.0
data_mix_in_col_dict: {'user': 'instruction', 'chip2': 'output'}
data_mix_in_prompt_type: instruct
output_dir: pythia-6.9b.openassistant_oasst1.json.1_epochs.60adcad5963c6c9f0dfa1a69610159ad80436095.14
lora_weights: 
batch_size: 128
micro_batch_size: 2
gradient_checkpointing: False
fp16: True
num_epochs: 1
learning_rate: 0.0003
val_set_size: None
val_metrics: []
eval_steps: None
eval_epochs: None
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: None
llama_type: False
group_by_length: False
resume_from_checkpoint: pythia-6.9b.openassistant_oasst1.json.1_epochs.04664ddb38effe9618af5d5c7c32f8fa3706ef0f.14/checkpoint-342
ddp: True
local_files_only: False
resume_download: True
warmup_steps: 100
logging_steps: 1
save_steps: None
add_eos_token: False
world_size: 4
local_rank: 0
rank: 0
gpus: 4
device_map: auto
gradient_accumulation_steps: 64
cutoff_len: 512
prompt_type: plain
train_on_inputs: True
Command: finetune.py --base_model=EleutherAI/pythia-6.9b --data_path=openassistant_oasst1.json --prompt_type=plain --run_id=14 --cutoff_len=512 --resume_from_checkpoint=pythia-6.9b.openassistant_oasst1.json.1_epochs.04664ddb38effe9618af5d5c7c32f8fa3706ef0f.14/checkpoint-342 --micro_batch_size=2
Hash: 60adcad5963c6c9f0dfa1a69610159ad80436095
Distributed: data parallel

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:05<00:05,  5.13s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:07<00:07,  7.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:07<00:00,  3.28s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:07<00:00,  3.56s/it]
Checkpoint pythia-6.9b.openassistant_oasst1.json.1_epochs.04664ddb38effe9618af5d5c7c32f8fa3706ef0f.14/checkpoint-342/adapter_model.bin not found
PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50432, 4096)
        (layers): ModuleList(
          (0-31): 32 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=4096, out_features=12288, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=4096, out_features=16, bias=False)
                (lora_B): Conv1d(16, 8192, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=4096, out_features=4096, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=4096, out_features=16384, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=16384, out_features=4096, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=4096, out_features=50432, bias=False)
    )
  )
)
trainable params: 4194304 || all params: 6861496320 || trainable%: 0.06112812430977155
Using Validation Metrics: []
Supported Metrics: ['bleu', 'rouge', 'sacrebleu', 'meteor']
Auto set val_set_size 1000
Found cached dataset json (/home/arno/.cache/huggingface/datasets/json/default-cdbbb11a140d3908/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 753.29it/s]

Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=16):   0%|          | 1/46283 [00:00<2:07:09,  6.07 examples/s]
Map (num_proc=16):   1%|          | 444/46283 [00:00<00:23, 1912.84 examples/s]
Map (num_proc=16):   4%|▍         | 1899/46283 [00:00<00:06, 6629.24 examples/s]
Map (num_proc=16):   8%|▊         | 3918/46283 [00:00<00:03, 11322.58 examples/s]
Map (num_proc=16):  13%|█▎        | 6131/46283 [00:00<00:02, 14815.42 examples/s]
Map (num_proc=16):  18%|█▊        | 8402/46283 [00:00<00:02, 17333.97 examples/s]
Map (num_proc=16):  23%|██▎       | 10585/46283 [00:00<00:01, 18736.67 examples/s]
Map (num_proc=16):  27%|██▋       | 12579/46283 [00:00<00:02, 16704.98 examples/s]
Map (num_proc=16):  31%|███       | 14358/46283 [00:01<00:01, 16883.84 examples/s]
Map (num_proc=16):  35%|███▌      | 16325/46283 [00:01<00:01, 17480.02 examples/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:09<00:00,  4.36s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:09<00:00,  4.77s/it]

Map (num_proc=16):  40%|███▉      | 18501/46283 [00:01<00:01, 18669.34 examples/s]
Map (num_proc=16):  44%|████▍     | 20409/46283 [00:01<00:01, 18549.55 examples/s]
Map (num_proc=16):  48%|████▊     | 22327/46283 [00:01<00:01, 18639.73 examples/s]
Map (num_proc=16):  52%|█████▏    | 24265/46283 [00:01<00:01, 17512.36 examples/s]
Map (num_proc=16):  56%|█████▋    | 26092/46283 [00:01<00:01, 17257.31 examples/s]PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50432, 4096)
        (layers): ModuleList(
          (0-31): 32 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=4096, out_features=12288, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=4096, out_features=16, bias=False)
                (lora_B): Conv1d(16, 8192, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=4096, out_features=4096, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=4096, out_features=16384, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=16384, out_features=4096, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=4096, out_features=50432, bias=False)
    )
  )
)
trainable params: 4194304 || all params: 6861496320 || trainable%: 0.06112812430977155

Map (num_proc=16):  61%|██████    | 28106/46283 [00:01<00:01, 18041.57 examples/s]
Map (num_proc=16):  66%|██████▌   | 30415/46283 [00:01<00:00, 19407.34 examples/s]
Map (num_proc=16):  71%|███████   | 32899/46283 [00:01<00:00, 20945.43 examples/s]
Map (num_proc=16):  76%|███████▌  | 35019/46283 [00:02<00:00, 20949.90 examples/s]
Map (num_proc=16):  80%|████████  | 37133/46283 [00:02<00:00, 19132.38 examples/s]Found cached dataset json (/home/arno/.cache/huggingface/datasets/json/default-cdbbb11a140d3908/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 189.93it/s]

Map (num_proc=16):  85%|████████▍ | 39147/46283 [00:02<00:00, 15782.52 examples/s]
Map (num_proc=16):  88%|████████▊ | 40939/46283 [00:02<00:00, 15223.66 examples/s]
Map (num_proc=16):  92%|█████████▏| 42675/46283 [00:02<00:00, 15746.29 examples/s]
Map (num_proc=16):  96%|█████████▌| 44382/46283 [00:02<00:00, 15304.34 examples/s]
Map (num_proc=16):  99%|█████████▉| 46039/46283 [00:02<00:00, 12735.03 examples/s]
                                                                                  
Final fine-tuning data:
Train Dataset({
    features: ['input', 'prompt_type', 'source', 'input_ids', 'attention_mask', 'labels'],
    num_rows: 46283
})
Valid None
Sample input: {'input': ['<human>: Can you summarize Dostoevsky\'s "The House of the Dead" novel?\n<bot>: "The House of the Dead" is a semi-autobiographical novel by Fyodor Dostoevsky, first published in 1861. The novel is set in a Siberian prison camp and is based on Dostoevsky\'s own experiences as a political prisoner.\n\nThe protagonist, Alexis Petrovich, is a young nobleman who is sent to the prison camp for participating in a revolutionary organization. While there, he meets a diverse cast of characters, including hardened criminals, petty thieves, and political prisoners. Through his interactions with these characters, Dostoevsky explores the nature of crime and punishment, the role of the state, and the complexities of human relationships.\n\nThroughout the novel, Dostoevsky portrays the prison as a microcosm of society, where the prisoners are subject to brutal conditions, lack of privacy, and near-constant surveillance. Despite this, they still find ways to maintain their humanity, often forming close bonds with one another and developing a sense of community.\n\nIn the end, Alexis is released from prison and returns to society, where he is torn between his experiences in the prison camp and his desire to reintegrate into the upper-class society he came from. "The House of the Dead" is a powerful meditation on the themes of crime and punishment, human nature, and the impact of societal institutions on the individual.\n\n<human>: Could you summarize that in a way that a 5-year-old could understand?\n'], 'prompt_type': ['plain'], 'source': ['OpenAssistant/oasst1']}
Auto set eval_steps to 18 out of 361 total training steps
Auto step save_steps to 18

Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=16):   0%|          | 1/46283 [00:00<1:46:53,  7.22 examples/s]
Map (num_proc=16):   1%|          | 340/46283 [00:00<00:28, 1602.43 examples/s]
Map (num_proc=16):   3%|▎         | 1577/46283 [00:00<00:07, 5935.64 examples/s]
Map (num_proc=16):   8%|▊         | 3713/46283 [00:00<00:03, 11558.07 examples/s]
Map (num_proc=16):  13%|█▎        | 6082/46283 [00:00<00:02, 15663.99 examples/s]
Map (num_proc=16):  18%|█▊        | 8463/46283 [00:00<00:02, 18289.40 examples/s]
Map (num_proc=16):  23%|██▎       | 10723/46283 [00:00<00:01, 19625.24 examples/s]
Map (num_proc=16):  28%|██▊       | 12881/46283 [00:00<00:01, 20191.83 examples/s]
Map (num_proc=16):  32%|███▏      | 14955/46283 [00:00<00:01, 19676.19 examples/s]
Map (num_proc=16):  37%|███▋      | 16994/46283 [00:01<00:01, 18776.06 examples/s]
Map (num_proc=16):  41%|████▏     | 19114/46283 [00:01<00:01, 19425.72 examples/s]
Map (num_proc=16):  46%|████▌     | 21226/46283 [00:01<00:01, 19847.43 examples/s]
Map (num_proc=16):  50%|█████     | 23273/46283 [00:01<00:01, 18834.42 examples/s]
Map (num_proc=16):  54%|█████▍    | 25223/46283 [00:01<00:01, 18561.94 examples/s]
Map (num_proc=16):  59%|█████▉    | 27232/46283 [00:01<00:01, 18899.70 examples/s]
Map (num_proc=16):  63%|██████▎   | 29177/46283 [00:01<00:00, 18970.34 examples/s]
Map (num_proc=16):  68%|██████▊   | 31268/46283 [00:01<00:00, 19489.90 examples/s]
Map (num_proc=16):  72%|███████▏  | 33456/46283 [00:01<00:00, 20182.00 examples/s]
Map (num_proc=16):  77%|███████▋  | 35530/46283 [00:02<00:00, 19330.61 examples/s]
Map (num_proc=16):  81%|████████  | 37497/46283 [00:02<00:00, 18616.52 examples/s]
Map (num_proc=16):  85%|████████▌ | 39372/46283 [00:02<00:00, 16264.68 examples/s]
Map (num_proc=16):  89%|████████▉ | 41088/46283 [00:02<00:00, 14381.10 examples/s]
Map (num_proc=16):  93%|█████████▎| 42832/46283 [00:02<00:00, 15078.45 examples/s]
Map (num_proc=16):  96%|█████████▌| 44418/46283 [00:02<00:00, 13787.50 examples/s]
Map (num_proc=16):  99%|█████████▉| 45857/46283 [00:02<00:00, 12094.10 examples/s]
                                                                                  
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

  0%|          | 0/361 [00:00<?, ?it/s]You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)

  0%|          | 1/361 [00:28<2:48:55, 28.16s/it]
                                                 
{'loss': 1.9411, 'learning_rate': 2.9999999999999997e-06, 'epoch': 0.0}

  0%|          | 1/361 [00:28<2:48:55, 28.16s/it]
  1%|          | 2/361 [00:52<2:35:37, 26.01s/it]
                                                 
{'loss': 1.9996, 'learning_rate': 5.999999999999999e-06, 'epoch': 0.01}

  1%|          | 2/361 [00:52<2:35:37, 26.01s/it]
  1%|          | 3/361 [01:16<2:29:50, 25.11s/it]
                                                 
{'loss': 2.0271, 'learning_rate': 8.999999999999999e-06, 'epoch': 0.01}

  1%|          | 3/361 [01:16<2:29:50, 25.11s/it]
  1%|          | 4/361 [01:40<2:27:13, 24.75s/it]
                                                 
{'loss': 1.9523, 'learning_rate': 1.1999999999999999e-05, 'epoch': 0.01}

  1%|          | 4/361 [01:40<2:27:13, 24.75s/it]
  1%|▏         | 5/361 [02:05<2:25:36, 24.54s/it]
                                                 
{'loss': 1.9739, 'learning_rate': 1.4999999999999999e-05, 'epoch': 0.01}

  1%|▏         | 5/361 [02:05<2:25:36, 24.54s/it]
  2%|▏         | 6/361 [02:28<2:22:46, 24.13s/it]
                                                 
{'loss': 2.0253, 'learning_rate': 1.7999999999999997e-05, 'epoch': 0.02}

  2%|▏         | 6/361 [02:28<2:22:46, 24.13s/it]
  2%|▏         | 7/361 [02:52<2:23:08, 24.26s/it]
                                                 
{'loss': 1.9403, 'learning_rate': 2.1e-05, 'epoch': 0.02}

  2%|▏         | 7/361 [02:52<2:23:08, 24.26s/it]
  2%|▏         | 8/361 [03:18<2:24:16, 24.52s/it]
                                                 
{'loss': 1.9508, 'learning_rate': 2.3999999999999997e-05, 'epoch': 0.02}

  2%|▏         | 8/361 [03:18<2:24:16, 24.52s/it]
  2%|▏         | 9/361 [03:43<2:25:30, 24.80s/it]
                                                 
{'loss': 1.923, 'learning_rate': 2.6999999999999996e-05, 'epoch': 0.02}

  2%|▏         | 9/361 [03:43<2:25:30, 24.80s/it]
  3%|▎         | 10/361 [04:08<2:25:59, 24.95s/it]
                                                  
{'loss': 1.9255, 'learning_rate': 2.9999999999999997e-05, 'epoch': 0.03}

  3%|▎         | 10/361 [04:08<2:25:59, 24.95s/it]
  3%|▎         | 11/361 [04:34<2:27:04, 25.21s/it]
                                                  
{'loss': 1.8328, 'learning_rate': 3.2999999999999996e-05, 'epoch': 0.03}

  3%|▎         | 11/361 [04:34<2:27:04, 25.21s/it]
  3%|▎         | 12/361 [04:59<2:26:37, 25.21s/it]
                                                  
{'loss': 1.929, 'learning_rate': 3.5999999999999994e-05, 'epoch': 0.03}

  3%|▎         | 12/361 [04:59<2:26:37, 25.21s/it]
  4%|▎         | 13/361 [05:26<2:29:13, 25.73s/it]
                                                  
{'loss': 1.9037, 'learning_rate': 3.9e-05, 'epoch': 0.04}

  4%|▎         | 13/361 [05:26<2:29:13, 25.73s/it]
  4%|▍         | 14/361 [05:50<2:25:52, 25.22s/it]
                                                  
{'loss': 1.9376, 'learning_rate': 4.2e-05, 'epoch': 0.04}

  4%|▍         | 14/361 [05:50<2:25:52, 25.22s/it]
  4%|▍         | 15/361 [06:14<2:22:21, 24.69s/it]
                                                  
{'loss': 1.9225, 'learning_rate': 4.4999999999999996e-05, 'epoch': 0.04}

  4%|▍         | 15/361 [06:14<2:22:21, 24.69s/it]
  4%|▍         | 16/361 [06:37<2:20:14, 24.39s/it]
                                                  
{'loss': 1.8579, 'learning_rate': 4.7999999999999994e-05, 'epoch': 0.04}

  4%|▍         | 16/361 [06:37<2:20:14, 24.39s/it]
  5%|▍         | 17/361 [07:02<2:21:01, 24.60s/it]
                                                  
{'loss': 1.9508, 'learning_rate': 5.1e-05, 'epoch': 0.05}

  5%|▍         | 17/361 [07:02<2:21:01, 24.60s/it]
  5%|▍         | 18/361 [07:28<2:21:50, 24.81s/it]
                                                  
{'loss': 1.8489, 'learning_rate': 5.399999999999999e-05, 'epoch': 0.05}

  5%|▍         | 18/361 [07:28<2:21:50, 24.81s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

  5%|▌         | 19/361 [07:54<2:23:49, 25.23s/it]
                                                  
{'loss': 1.7821, 'learning_rate': 5.6999999999999996e-05, 'epoch': 0.05}

  5%|▌         | 19/361 [07:54<2:23:49, 25.23s/it]
  6%|▌         | 20/361 [08:20<2:24:36, 25.44s/it]
                                                  
{'loss': 1.7501, 'learning_rate': 5.9999999999999995e-05, 'epoch': 0.06}

  6%|▌         | 20/361 [08:20<2:24:36, 25.44s/it]
  6%|▌         | 21/361 [08:45<2:24:08, 25.44s/it]
                                                  
{'loss': 1.876, 'learning_rate': 6.299999999999999e-05, 'epoch': 0.06}

  6%|▌         | 21/361 [08:45<2:24:08, 25.44s/it]
  6%|▌         | 22/361 [09:11<2:24:16, 25.53s/it]
                                                  
{'loss': 1.8552, 'learning_rate': 6.599999999999999e-05, 'epoch': 0.06}

  6%|▌         | 22/361 [09:11<2:24:16, 25.53s/it]
  6%|▋         | 23/361 [09:37<2:24:52, 25.72s/it]
                                                  
{'loss': 1.7418, 'learning_rate': 6.9e-05, 'epoch': 0.06}

  6%|▋         | 23/361 [09:37<2:24:52, 25.72s/it]
  7%|▋         | 24/361 [10:02<2:22:54, 25.44s/it]
                                                  
{'loss': 1.7396, 'learning_rate': 7.199999999999999e-05, 'epoch': 0.07}

  7%|▋         | 24/361 [10:02<2:22:54, 25.44s/it]
  7%|▋         | 25/361 [10:28<2:23:25, 25.61s/it]
                                                  
{'loss': 1.8312, 'learning_rate': 7.5e-05, 'epoch': 0.07}

  7%|▋         | 25/361 [10:28<2:23:25, 25.61s/it]
  7%|▋         | 26/361 [10:53<2:21:20, 25.32s/it]
                                                  
{'loss': 1.8323, 'learning_rate': 7.8e-05, 'epoch': 0.07}

  7%|▋         | 26/361 [10:53<2:21:20, 25.32s/it]
  7%|▋         | 27/361 [11:18<2:21:05, 25.34s/it]
                                                  
{'loss': 1.7739, 'learning_rate': 8.1e-05, 'epoch': 0.07}

  7%|▋         | 27/361 [11:18<2:21:05, 25.34s/it]
  8%|▊         | 28/361 [11:43<2:20:05, 25.24s/it]
                                                  
{'loss': 1.8432, 'learning_rate': 8.4e-05, 'epoch': 0.08}

  8%|▊         | 28/361 [11:43<2:20:05, 25.24s/it]
  8%|▊         | 29/361 [12:10<2:22:51, 25.82s/it]
                                                  
{'loss': 1.795, 'learning_rate': 8.699999999999999e-05, 'epoch': 0.08}

  8%|▊         | 29/361 [12:10<2:22:51, 25.82s/it]
  8%|▊         | 30/361 [12:36<2:22:38, 25.86s/it]
                                                  
{'loss': 1.6552, 'learning_rate': 8.999999999999999e-05, 'epoch': 0.08}

  8%|▊         | 30/361 [12:36<2:22:38, 25.86s/it]
  9%|▊         | 31/361 [13:01<2:20:21, 25.52s/it]
                                                  
{'loss': 1.6393, 'learning_rate': 9.3e-05, 'epoch': 0.09}

  9%|▊         | 31/361 [13:01<2:20:21, 25.52s/it]
  9%|▉         | 32/361 [13:26<2:19:44, 25.49s/it]
                                                  
{'loss': 1.7494, 'learning_rate': 9.599999999999999e-05, 'epoch': 0.09}

  9%|▉         | 32/361 [13:27<2:19:44, 25.49s/it]
  9%|▉         | 33/361 [13:52<2:19:04, 25.44s/it]
                                                  
{'loss': 1.7987, 'learning_rate': 9.9e-05, 'epoch': 0.09}

  9%|▉         | 33/361 [13:52<2:19:04, 25.44s/it]
  9%|▉         | 34/361 [14:17<2:18:45, 25.46s/it]
                                                  
{'loss': 1.6861, 'learning_rate': 0.000102, 'epoch': 0.09}

  9%|▉         | 34/361 [14:17<2:18:45, 25.46s/it]
 10%|▉         | 35/361 [14:43<2:19:27, 25.67s/it]
                                                  
{'loss': 1.7583, 'learning_rate': 0.00010499999999999999, 'epoch': 0.1}

 10%|▉         | 35/361 [14:43<2:19:27, 25.67s/it]
 10%|▉         | 36/361 [15:09<2:19:03, 25.67s/it]
                                                  
{'loss': 1.7063, 'learning_rate': 0.00010799999999999998, 'epoch': 0.1}

 10%|▉         | 36/361 [15:09<2:19:03, 25.67s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 10%|█         | 37/361 [15:34<2:17:54, 25.54s/it]
                                                  
{'loss': 1.7649, 'learning_rate': 0.00011099999999999999, 'epoch': 0.1}

 10%|█         | 37/361 [15:34<2:17:54, 25.54s/it]
 11%|█         | 38/361 [16:00<2:17:18, 25.51s/it]
                                                  
{'loss': 1.7365, 'learning_rate': 0.00011399999999999999, 'epoch': 0.11}

 11%|█         | 38/361 [16:00<2:17:18, 25.51s/it]
 11%|█         | 39/361 [16:24<2:14:45, 25.11s/it]
                                                  
{'loss': 1.6391, 'learning_rate': 0.000117, 'epoch': 0.11}

 11%|█         | 39/361 [16:24<2:14:45, 25.11s/it]
 11%|█         | 40/361 [16:49<2:13:53, 25.03s/it]
                                                  
{'loss': 1.639, 'learning_rate': 0.00011999999999999999, 'epoch': 0.11}

 11%|█         | 40/361 [16:49<2:13:53, 25.03s/it]
 11%|█▏        | 41/361 [17:14<2:14:07, 25.15s/it]
                                                  
{'loss': 1.7422, 'learning_rate': 0.00012299999999999998, 'epoch': 0.11}

 11%|█▏        | 41/361 [17:14<2:14:07, 25.15s/it]
 12%|█▏        | 42/361 [17:39<2:13:31, 25.11s/it]
                                                  
{'loss': 1.6633, 'learning_rate': 0.00012599999999999997, 'epoch': 0.12}

 12%|█▏        | 42/361 [17:39<2:13:31, 25.11s/it]
 12%|█▏        | 43/361 [18:04<2:12:32, 25.01s/it]
                                                  
{'loss': 1.7144, 'learning_rate': 0.000129, 'epoch': 0.12}

 12%|█▏        | 43/361 [18:04<2:12:32, 25.01s/it]
 12%|█▏        | 44/361 [18:30<2:13:46, 25.32s/it]
                                                  
{'loss': 1.6037, 'learning_rate': 0.00013199999999999998, 'epoch': 0.12}

 12%|█▏        | 44/361 [18:30<2:13:46, 25.32s/it]
 12%|█▏        | 45/361 [18:53<2:10:03, 24.70s/it]
                                                  
{'loss': 1.6832, 'learning_rate': 0.000135, 'epoch': 0.12}

 12%|█▏        | 45/361 [18:53<2:10:03, 24.70s/it]
 13%|█▎        | 46/361 [19:16<2:07:17, 24.25s/it]
                                                  
{'loss': 1.6703, 'learning_rate': 0.000138, 'epoch': 0.13}

 13%|█▎        | 46/361 [19:16<2:07:17, 24.25s/it]
 13%|█▎        | 47/361 [19:43<2:11:11, 25.07s/it]
                                                  
{'loss': 1.6936, 'learning_rate': 0.00014099999999999998, 'epoch': 0.13}

 13%|█▎        | 47/361 [19:43<2:11:11, 25.07s/it]
 13%|█▎        | 48/361 [20:07<2:09:13, 24.77s/it]
                                                  
{'loss': 1.6507, 'learning_rate': 0.00014399999999999998, 'epoch': 0.13}

 13%|█▎        | 48/361 [20:07<2:09:13, 24.77s/it]
 14%|█▎        | 49/361 [20:33<2:09:46, 24.96s/it]
                                                  
{'loss': 1.5924, 'learning_rate': 0.000147, 'epoch': 0.14}

 14%|█▎        | 49/361 [20:33<2:09:46, 24.96s/it]
 14%|█▍        | 50/361 [20:58<2:10:20, 25.15s/it]
                                                  
{'loss': 1.6229, 'learning_rate': 0.00015, 'epoch': 0.14}

 14%|█▍        | 50/361 [20:59<2:10:20, 25.15s/it]
 14%|█▍        | 51/361 [21:25<2:11:49, 25.51s/it]
                                                  
{'loss': 1.6524, 'learning_rate': 0.00015299999999999998, 'epoch': 0.14}

 14%|█▍        | 51/361 [21:25<2:11:49, 25.51s/it]
 14%|█▍        | 52/361 [21:50<2:11:02, 25.44s/it]
                                                  
{'loss': 1.5887, 'learning_rate': 0.000156, 'epoch': 0.14}

 14%|█▍        | 52/361 [21:50<2:11:02, 25.44s/it]
 15%|█▍        | 53/361 [22:15<2:09:17, 25.19s/it]
                                                  
{'loss': 1.5967, 'learning_rate': 0.000159, 'epoch': 0.15}

 15%|█▍        | 53/361 [22:15<2:09:17, 25.19s/it]
 15%|█▍        | 54/361 [22:40<2:09:17, 25.27s/it]
                                                  
{'loss': 1.6306, 'learning_rate': 0.000162, 'epoch': 0.15}

 15%|█▍        | 54/361 [22:40<2:09:17, 25.27s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 15%|█▌        | 55/361 [23:06<2:09:52, 25.47s/it]
                                                  
{'loss': 1.6323, 'learning_rate': 0.000165, 'epoch': 0.15}

 15%|█▌        | 55/361 [23:06<2:09:52, 25.47s/it]
 16%|█▌        | 56/361 [23:30<2:07:03, 24.99s/it]
                                                  
{'loss': 1.6364, 'learning_rate': 0.000168, 'epoch': 0.15}

 16%|█▌        | 56/361 [23:30<2:07:03, 24.99s/it]
 16%|█▌        | 57/361 [23:56<2:07:35, 25.18s/it]
                                                  
{'loss': 1.6355, 'learning_rate': 0.00017099999999999998, 'epoch': 0.16}

 16%|█▌        | 57/361 [23:56<2:07:35, 25.18s/it]
 16%|█▌        | 58/361 [24:21<2:06:54, 25.13s/it]
                                                  
{'loss': 1.6074, 'learning_rate': 0.00017399999999999997, 'epoch': 0.16}

 16%|█▌        | 58/361 [24:21<2:06:54, 25.13s/it]
 16%|█▋        | 59/361 [24:45<2:05:36, 24.95s/it]
                                                  
{'loss': 1.6492, 'learning_rate': 0.00017699999999999997, 'epoch': 0.16}

 16%|█▋        | 59/361 [24:45<2:05:36, 24.95s/it]
 17%|█▋        | 60/361 [25:11<2:05:54, 25.10s/it]
                                                  
{'loss': 1.5707, 'learning_rate': 0.00017999999999999998, 'epoch': 0.17}

 17%|█▋        | 60/361 [25:11<2:05:54, 25.10s/it]
 17%|█▋        | 61/361 [25:37<2:07:25, 25.48s/it]
                                                  
{'loss': 1.5497, 'learning_rate': 0.00018299999999999998, 'epoch': 0.17}

 17%|█▋        | 61/361 [25:37<2:07:25, 25.48s/it]
 17%|█▋        | 62/361 [26:02<2:05:54, 25.27s/it]
                                                  
{'loss': 1.6316, 'learning_rate': 0.000186, 'epoch': 0.17}

 17%|█▋        | 62/361 [26:02<2:05:54, 25.27s/it]
 17%|█▋        | 63/361 [26:27<2:05:18, 25.23s/it]
                                                  
{'loss': 1.6221, 'learning_rate': 0.00018899999999999999, 'epoch': 0.17}

 17%|█▋        | 63/361 [26:27<2:05:18, 25.23s/it]
 18%|█▊        | 64/361 [26:50<2:02:09, 24.68s/it]
                                                  
{'loss': 1.6293, 'learning_rate': 0.00019199999999999998, 'epoch': 0.18}

 18%|█▊        | 64/361 [26:50<2:02:09, 24.68s/it]
 18%|█▊        | 65/361 [27:15<2:01:50, 24.70s/it]
                                                  
{'loss': 1.614, 'learning_rate': 0.000195, 'epoch': 0.18}

 18%|█▊        | 65/361 [27:15<2:01:50, 24.70s/it]
 18%|█▊        | 66/361 [27:40<2:02:34, 24.93s/it]
                                                  
{'loss': 1.5383, 'learning_rate': 0.000198, 'epoch': 0.18}

 18%|█▊        | 66/361 [27:40<2:02:34, 24.93s/it]
 19%|█▊        | 67/361 [28:05<2:01:14, 24.74s/it]
                                                  
{'loss': 1.6067, 'learning_rate': 0.000201, 'epoch': 0.19}

 19%|█▊        | 67/361 [28:05<2:01:14, 24.74s/it]
 19%|█▉        | 68/361 [28:30<2:01:55, 24.97s/it]
                                                  
{'loss': 1.5303, 'learning_rate': 0.000204, 'epoch': 0.19}

 19%|█▉        | 68/361 [28:30<2:01:55, 24.97s/it]
 19%|█▉        | 69/361 [28:55<2:01:07, 24.89s/it]
                                                  
{'loss': 1.5979, 'learning_rate': 0.00020699999999999996, 'epoch': 0.19}

 19%|█▉        | 69/361 [28:55<2:01:07, 24.89s/it]
 19%|█▉        | 70/361 [29:21<2:02:32, 25.27s/it]
                                                  
{'loss': 1.6423, 'learning_rate': 0.00020999999999999998, 'epoch': 0.19}

 19%|█▉        | 70/361 [29:21<2:02:32, 25.27s/it]
 20%|█▉        | 71/361 [29:47<2:02:26, 25.33s/it]
                                                  
{'loss': 1.5637, 'learning_rate': 0.00021299999999999997, 'epoch': 0.2}

 20%|█▉        | 71/361 [29:47<2:02:26, 25.33s/it]
 20%|█▉        | 72/361 [30:11<2:00:35, 25.04s/it]
                                                  
{'loss': 1.6178, 'learning_rate': 0.00021599999999999996, 'epoch': 0.2}

 20%|█▉        | 72/361 [30:11<2:00:35, 25.04s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 20%|██        | 73/361 [30:37<2:01:07, 25.23s/it]
                                                  
{'loss': 1.5735, 'learning_rate': 0.00021899999999999998, 'epoch': 0.2}

 20%|██        | 73/361 [30:37<2:01:07, 25.23s/it]
 20%|██        | 74/361 [31:01<2:00:00, 25.09s/it]
                                                  
{'loss': 1.592, 'learning_rate': 0.00022199999999999998, 'epoch': 0.2}

 20%|██        | 74/361 [31:01<2:00:00, 25.09s/it]
 21%|██        | 75/361 [31:26<1:59:26, 25.06s/it]
                                                  
{'loss': 1.6033, 'learning_rate': 0.000225, 'epoch': 0.21}

 21%|██        | 75/361 [31:26<1:59:26, 25.06s/it]
 21%|██        | 76/361 [31:51<1:59:09, 25.09s/it]
                                                  
{'loss': 1.6805, 'learning_rate': 0.00022799999999999999, 'epoch': 0.21}

 21%|██        | 76/361 [31:52<1:59:09, 25.09s/it]
 21%|██▏       | 77/361 [32:16<1:58:07, 24.96s/it]
                                                  
{'loss': 1.6224, 'learning_rate': 0.00023099999999999998, 'epoch': 0.21}

 21%|██▏       | 77/361 [32:16<1:58:07, 24.96s/it]
 22%|██▏       | 78/361 [32:40<1:56:43, 24.75s/it]
                                                  
{'loss': 1.6509, 'learning_rate': 0.000234, 'epoch': 0.22}

 22%|██▏       | 78/361 [32:40<1:56:43, 24.75s/it]
 22%|██▏       | 79/361 [33:06<1:57:23, 24.98s/it]
                                                  
{'loss': 1.6461, 'learning_rate': 0.000237, 'epoch': 0.22}

 22%|██▏       | 79/361 [33:06<1:57:23, 24.98s/it]
 22%|██▏       | 80/361 [33:31<1:56:32, 24.89s/it]
                                                  
{'loss': 1.6474, 'learning_rate': 0.00023999999999999998, 'epoch': 0.22}

 22%|██▏       | 80/361 [33:31<1:56:32, 24.89s/it]
 22%|██▏       | 81/361 [33:56<1:56:43, 25.01s/it]
                                                  
{'loss': 1.578, 'learning_rate': 0.000243, 'epoch': 0.22}

 22%|██▏       | 81/361 [33:56<1:56:43, 25.01s/it]
 23%|██▎       | 82/361 [34:20<1:54:57, 24.72s/it]
                                                  
{'loss': 1.6053, 'learning_rate': 0.00024599999999999996, 'epoch': 0.23}

 23%|██▎       | 82/361 [34:20<1:54:57, 24.72s/it]
 23%|██▎       | 83/361 [34:45<1:55:13, 24.87s/it]
                                                  
{'loss': 1.6497, 'learning_rate': 0.000249, 'epoch': 0.23}

 23%|██▎       | 83/361 [34:45<1:55:13, 24.87s/it]
 23%|██▎       | 84/361 [35:11<1:56:08, 25.16s/it]
                                                  
{'loss': 1.5123, 'learning_rate': 0.00025199999999999995, 'epoch': 0.23}

 23%|██▎       | 84/361 [35:11<1:56:08, 25.16s/it]
 24%|██▎       | 85/361 [35:36<1:55:24, 25.09s/it]
                                                  
{'loss': 1.6291, 'learning_rate': 0.00025499999999999996, 'epoch': 0.24}

 24%|██▎       | 85/361 [35:36<1:55:24, 25.09s/it]
 24%|██▍       | 86/361 [36:00<1:54:09, 24.91s/it]
                                                  
{'loss': 1.6438, 'learning_rate': 0.000258, 'epoch': 0.24}

 24%|██▍       | 86/361 [36:00<1:54:09, 24.91s/it]
 24%|██▍       | 87/361 [36:25<1:52:39, 24.67s/it]
                                                  
{'loss': 1.5996, 'learning_rate': 0.000261, 'epoch': 0.24}

 24%|██▍       | 87/361 [36:25<1:52:39, 24.67s/it]
 24%|██▍       | 88/361 [36:50<1:53:04, 24.85s/it]
                                                  
{'loss': 1.6084, 'learning_rate': 0.00026399999999999997, 'epoch': 0.24}

 24%|██▍       | 88/361 [36:50<1:53:04, 24.85s/it]
 25%|██▍       | 89/361 [37:15<1:52:53, 24.90s/it]
                                                  
{'loss': 1.6086, 'learning_rate': 0.000267, 'epoch': 0.25}

 25%|██▍       | 89/361 [37:15<1:52:53, 24.90s/it]
 25%|██▍       | 90/361 [37:40<1:53:19, 25.09s/it]
                                                  
{'loss': 1.5686, 'learning_rate': 0.00027, 'epoch': 0.25}

 25%|██▍       | 90/361 [37:40<1:53:19, 25.09s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 25%|██▌       | 91/361 [38:07<1:54:28, 25.44s/it]
                                                  
{'loss': 1.6887, 'learning_rate': 0.00027299999999999997, 'epoch': 0.25}

 25%|██▌       | 91/361 [38:07<1:54:28, 25.44s/it]
 25%|██▌       | 92/361 [38:32<1:54:15, 25.49s/it]
                                                  
{'loss': 1.6227, 'learning_rate': 0.000276, 'epoch': 0.25}

 25%|██▌       | 92/361 [38:32<1:54:15, 25.49s/it]
 26%|██▌       | 93/361 [38:57<1:53:06, 25.32s/it]
                                                  
{'loss': 1.5497, 'learning_rate': 0.000279, 'epoch': 0.26}

 26%|██▌       | 93/361 [38:57<1:53:06, 25.32s/it]
 26%|██▌       | 94/361 [39:22<1:52:17, 25.23s/it]
                                                  
{'loss': 1.5682, 'learning_rate': 0.00028199999999999997, 'epoch': 0.26}

 26%|██▌       | 94/361 [39:22<1:52:17, 25.23s/it]
 26%|██▋       | 95/361 [39:48<1:53:08, 25.52s/it]
                                                  
{'loss': 1.5042, 'learning_rate': 0.000285, 'epoch': 0.26}

 26%|██▋       | 95/361 [39:48<1:53:08, 25.52s/it]
 27%|██▋       | 96/361 [40:13<1:50:59, 25.13s/it]
                                                  
{'loss': 1.5936, 'learning_rate': 0.00028799999999999995, 'epoch': 0.27}

 27%|██▋       | 96/361 [40:13<1:50:59, 25.13s/it]
 27%|██▋       | 97/361 [40:39<1:52:00, 25.46s/it]
                                                  
{'loss': 1.5927, 'learning_rate': 0.00029099999999999997, 'epoch': 0.27}

 27%|██▋       | 97/361 [40:39<1:52:00, 25.46s/it]
 27%|██▋       | 98/361 [41:03<1:50:31, 25.21s/it]
                                                  
{'loss': 1.5903, 'learning_rate': 0.000294, 'epoch': 0.27}

 27%|██▋       | 98/361 [41:03<1:50:31, 25.21s/it]
 27%|██▋       | 99/361 [41:29<1:50:48, 25.38s/it]
                                                  
{'loss': 1.6368, 'learning_rate': 0.00029699999999999996, 'epoch': 0.27}

 27%|██▋       | 99/361 [41:29<1:50:48, 25.38s/it]
 28%|██▊       | 100/361 [41:55<1:51:11, 25.56s/it]
                                                   
{'loss': 1.6028, 'learning_rate': 0.0003, 'epoch': 0.28}

 28%|██▊       | 100/361 [41:55<1:51:11, 25.56s/it]
 28%|██▊       | 101/361 [42:20<1:49:48, 25.34s/it]
                                                   
{'loss': 1.6066, 'learning_rate': 0.00029885057471264366, 'epoch': 0.28}

 28%|██▊       | 101/361 [42:20<1:49:48, 25.34s/it]
 28%|██▊       | 102/361 [42:46<1:49:49, 25.44s/it]
                                                   
{'loss': 1.5615, 'learning_rate': 0.00029770114942528734, 'epoch': 0.28}

 28%|██▊       | 102/361 [42:46<1:49:49, 25.44s/it]
 29%|██▊       | 103/361 [43:11<1:48:49, 25.31s/it]
                                                   
{'loss': 1.5553, 'learning_rate': 0.00029655172413793103, 'epoch': 0.28}

 29%|██▊       | 103/361 [43:11<1:48:49, 25.31s/it]
 29%|██▉       | 104/361 [43:36<1:48:27, 25.32s/it]
                                                   
{'loss': 1.6364, 'learning_rate': 0.00029540229885057466, 'epoch': 0.29}

 29%|██▉       | 104/361 [43:36<1:48:27, 25.32s/it]
 29%|██▉       | 105/361 [44:00<1:46:11, 24.89s/it]
                                                   
{'loss': 1.5682, 'learning_rate': 0.00029425287356321835, 'epoch': 0.29}

 29%|██▉       | 105/361 [44:00<1:46:11, 24.89s/it]
 29%|██▉       | 106/361 [44:26<1:46:41, 25.10s/it]
                                                   
{'loss': 1.5774, 'learning_rate': 0.00029310344827586203, 'epoch': 0.29}

 29%|██▉       | 106/361 [44:26<1:46:41, 25.10s/it]
 30%|██▉       | 107/361 [44:50<1:45:55, 25.02s/it]
                                                   
{'loss': 1.5784, 'learning_rate': 0.0002919540229885057, 'epoch': 0.3}

 30%|██▉       | 107/361 [44:50<1:45:55, 25.02s/it]
 30%|██▉       | 108/361 [45:16<1:46:24, 25.24s/it]
                                                   
{'loss': 1.5652, 'learning_rate': 0.0002908045977011494, 'epoch': 0.3}

 30%|██▉       | 108/361 [45:16<1:46:24, 25.24s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 30%|███       | 109/361 [45:39<1:43:16, 24.59s/it]
                                                   
{'loss': 1.6457, 'learning_rate': 0.0002896551724137931, 'epoch': 0.3}

 30%|███       | 109/361 [45:39<1:43:16, 24.59s/it]
 30%|███       | 110/361 [46:06<1:45:24, 25.20s/it]
                                                   
{'loss': 1.5875, 'learning_rate': 0.00028850574712643677, 'epoch': 0.3}

 30%|███       | 110/361 [46:06<1:45:24, 25.20s/it]
 31%|███       | 111/361 [46:32<1:46:03, 25.45s/it]
                                                   
{'loss': 1.6421, 'learning_rate': 0.0002873563218390804, 'epoch': 0.31}

 31%|███       | 111/361 [46:32<1:46:03, 25.45s/it]
 31%|███       | 112/361 [46:58<1:46:17, 25.61s/it]
                                                   
{'loss': 1.5667, 'learning_rate': 0.0002862068965517241, 'epoch': 0.31}

 31%|███       | 112/361 [46:58<1:46:17, 25.61s/it]
 31%|███▏      | 113/361 [47:23<1:44:47, 25.35s/it]
                                                   
{'loss': 1.6544, 'learning_rate': 0.00028505747126436777, 'epoch': 0.31}

 31%|███▏      | 113/361 [47:23<1:44:47, 25.35s/it]
 32%|███▏      | 114/361 [47:47<1:42:42, 24.95s/it]
                                                   
{'loss': 1.5575, 'learning_rate': 0.00028390804597701146, 'epoch': 0.32}

 32%|███▏      | 114/361 [47:47<1:42:42, 24.95s/it]
 32%|███▏      | 115/361 [48:12<1:42:40, 25.04s/it]
                                                   
{'loss': 1.5646, 'learning_rate': 0.00028275862068965514, 'epoch': 0.32}

 32%|███▏      | 115/361 [48:12<1:42:40, 25.04s/it]
 32%|███▏      | 116/361 [48:37<1:42:38, 25.14s/it]
                                                   
{'loss': 1.5621, 'learning_rate': 0.00028160919540229883, 'epoch': 0.32}

 32%|███▏      | 116/361 [48:37<1:42:38, 25.14s/it]
 32%|███▏      | 117/361 [49:01<1:40:30, 24.71s/it]
                                                   
{'loss': 1.6079, 'learning_rate': 0.0002804597701149425, 'epoch': 0.32}

 32%|███▏      | 117/361 [49:01<1:40:30, 24.71s/it]
 33%|███▎      | 118/361 [49:26<1:40:11, 24.74s/it]
                                                   
{'loss': 1.6032, 'learning_rate': 0.0002793103448275862, 'epoch': 0.33}

 33%|███▎      | 118/361 [49:26<1:40:11, 24.74s/it]
 33%|███▎      | 119/361 [49:50<1:39:18, 24.62s/it]
                                                   
{'loss': 1.6528, 'learning_rate': 0.0002781609195402299, 'epoch': 0.33}

 33%|███▎      | 119/361 [49:50<1:39:18, 24.62s/it]
 33%|███▎      | 120/361 [50:15<1:38:55, 24.63s/it]
                                                   
{'loss': 1.5986, 'learning_rate': 0.00027701149425287357, 'epoch': 0.33}

 33%|███▎      | 120/361 [50:15<1:38:55, 24.63s/it]
 34%|███▎      | 121/361 [50:40<1:39:11, 24.80s/it]
                                                   
{'loss': 1.5623, 'learning_rate': 0.0002758620689655172, 'epoch': 0.33}

 34%|███▎      | 121/361 [50:40<1:39:11, 24.80s/it]
 34%|███▍      | 122/361 [51:06<1:40:13, 25.16s/it]
                                                   
{'loss': 1.5869, 'learning_rate': 0.0002747126436781609, 'epoch': 0.34}

 34%|███▍      | 122/361 [51:06<1:40:13, 25.16s/it]
 34%|███▍      | 123/361 [51:31<1:39:30, 25.09s/it]
                                                   
{'loss': 1.5924, 'learning_rate': 0.00027356321839080457, 'epoch': 0.34}

 34%|███▍      | 123/361 [51:31<1:39:30, 25.09s/it]
 34%|███▍      | 124/361 [51:55<1:38:21, 24.90s/it]
                                                   
{'loss': 1.5255, 'learning_rate': 0.00027241379310344825, 'epoch': 0.34}

 34%|███▍      | 124/361 [51:55<1:38:21, 24.90s/it]
 35%|███▍      | 125/361 [52:21<1:38:20, 25.00s/it]
                                                   
{'loss': 1.5708, 'learning_rate': 0.00027126436781609194, 'epoch': 0.35}

 35%|███▍      | 125/361 [52:21<1:38:20, 25.00s/it]
 35%|███▍      | 126/361 [52:46<1:38:35, 25.17s/it]
                                                   
{'loss': 1.5733, 'learning_rate': 0.0002701149425287356, 'epoch': 0.35}

 35%|███▍      | 126/361 [52:46<1:38:35, 25.17s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 35%|███▌      | 127/361 [53:11<1:37:51, 25.09s/it]
                                                   
{'loss': 1.6024, 'learning_rate': 0.0002689655172413793, 'epoch': 0.35}

 35%|███▌      | 127/361 [53:11<1:37:51, 25.09s/it]
 35%|███▌      | 128/361 [53:37<1:38:54, 25.47s/it]
                                                   
{'loss': 1.6484, 'learning_rate': 0.00026781609195402294, 'epoch': 0.35}

 35%|███▌      | 128/361 [53:37<1:38:54, 25.47s/it]
 36%|███▌      | 129/361 [54:01<1:36:55, 25.07s/it]
                                                   
{'loss': 1.6066, 'learning_rate': 0.0002666666666666666, 'epoch': 0.36}

 36%|███▌      | 129/361 [54:02<1:36:55, 25.07s/it]
 36%|███▌      | 130/361 [54:27<1:37:23, 25.30s/it]
                                                   
{'loss': 1.5704, 'learning_rate': 0.0002655172413793103, 'epoch': 0.36}

 36%|███▌      | 130/361 [54:27<1:37:23, 25.30s/it]
 36%|███▋      | 131/361 [54:51<1:35:26, 24.90s/it]
                                                   
{'loss': 1.529, 'learning_rate': 0.000264367816091954, 'epoch': 0.36}

 36%|███▋      | 131/361 [54:51<1:35:26, 24.90s/it]
 37%|███▋      | 132/361 [55:16<1:34:45, 24.83s/it]
                                                   
{'loss': 1.5927, 'learning_rate': 0.0002632183908045977, 'epoch': 0.37}

 37%|███▋      | 132/361 [55:16<1:34:45, 24.83s/it]
 37%|███▋      | 133/361 [55:41<1:34:49, 24.95s/it]
                                                   
{'loss': 1.5594, 'learning_rate': 0.00026206896551724137, 'epoch': 0.37}

 37%|███▋      | 133/361 [55:41<1:34:49, 24.95s/it]
 37%|███▋      | 134/361 [56:05<1:33:34, 24.73s/it]
                                                   
{'loss': 1.5621, 'learning_rate': 0.00026091954022988505, 'epoch': 0.37}

 37%|███▋      | 134/361 [56:05<1:33:34, 24.73s/it]
 37%|███▋      | 135/361 [56:32<1:35:16, 25.29s/it]
                                                   
{'loss': 1.5752, 'learning_rate': 0.0002597701149425287, 'epoch': 0.37}

 37%|███▋      | 135/361 [56:32<1:35:16, 25.29s/it]
 38%|███▊      | 136/361 [56:58<1:35:19, 25.42s/it]
                                                   
{'loss': 1.5771, 'learning_rate': 0.00025862068965517237, 'epoch': 0.38}

 38%|███▊      | 136/361 [56:58<1:35:19, 25.42s/it]
 38%|███▊      | 137/361 [57:22<1:33:42, 25.10s/it]
                                                   
{'loss': 1.5151, 'learning_rate': 0.00025747126436781605, 'epoch': 0.38}

 38%|███▊      | 137/361 [57:22<1:33:42, 25.10s/it]
 38%|███▊      | 138/361 [57:47<1:33:29, 25.15s/it]
                                                   
{'loss': 1.6271, 'learning_rate': 0.00025632183908045974, 'epoch': 0.38}

 38%|███▊      | 138/361 [57:47<1:33:29, 25.15s/it]
 39%|███▊      | 139/361 [58:13<1:33:36, 25.30s/it]
                                                   
{'loss': 1.5311, 'learning_rate': 0.0002551724137931034, 'epoch': 0.38}

 39%|███▊      | 139/361 [58:13<1:33:36, 25.30s/it]
 39%|███▉      | 140/361 [58:38<1:33:18, 25.33s/it]
                                                   
{'loss': 1.5597, 'learning_rate': 0.0002540229885057471, 'epoch': 0.39}

 39%|███▉      | 140/361 [58:38<1:33:18, 25.33s/it]
 39%|███▉      | 141/361 [59:02<1:30:51, 24.78s/it]
                                                   
{'loss': 1.4951, 'learning_rate': 0.0002528735632183908, 'epoch': 0.39}

 39%|███▉      | 141/361 [59:02<1:30:51, 24.78s/it]
 39%|███▉      | 142/361 [59:28<1:31:38, 25.11s/it]
                                                   
{'loss': 1.6018, 'learning_rate': 0.0002517241379310345, 'epoch': 0.39}

 39%|███▉      | 142/361 [59:28<1:31:38, 25.11s/it]
 40%|███▉      | 143/361 [59:52<1:30:25, 24.89s/it]
                                                   
{'loss': 1.5442, 'learning_rate': 0.00025057471264367816, 'epoch': 0.4}

 40%|███▉      | 143/361 [59:52<1:30:25, 24.89s/it]
 40%|███▉      | 144/361 [1:00:16<1:29:04, 24.63s/it]
                                                     
{'loss': 1.6183, 'learning_rate': 0.00024942528735632185, 'epoch': 0.4}

 40%|███▉      | 144/361 [1:00:16<1:29:04, 24.63s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 40%|████      | 145/361 [1:00:43<1:30:39, 25.18s/it]
                                                     
{'loss': 1.6126, 'learning_rate': 0.0002482758620689655, 'epoch': 0.4}

 40%|████      | 145/361 [1:00:43<1:30:39, 25.18s/it]
 40%|████      | 146/361 [1:01:07<1:28:49, 24.79s/it]
                                                     
{'loss': 1.5384, 'learning_rate': 0.00024712643678160916, 'epoch': 0.4}

 40%|████      | 146/361 [1:01:07<1:28:49, 24.79s/it]
 41%|████      | 147/361 [1:01:32<1:28:51, 24.91s/it]
                                                     
{'loss': 1.5472, 'learning_rate': 0.00024597701149425285, 'epoch': 0.41}

 41%|████      | 147/361 [1:01:32<1:28:51, 24.91s/it]
 41%|████      | 148/361 [1:01:57<1:28:20, 24.89s/it]
                                                     
{'loss': 1.551, 'learning_rate': 0.00024482758620689653, 'epoch': 0.41}

 41%|████      | 148/361 [1:01:57<1:28:20, 24.89s/it]
 41%|████▏     | 149/361 [1:02:22<1:28:11, 24.96s/it]
                                                     
{'loss': 1.4971, 'learning_rate': 0.0002436781609195402, 'epoch': 0.41}

 41%|████▏     | 149/361 [1:02:22<1:28:11, 24.96s/it]
 42%|████▏     | 150/361 [1:02:48<1:28:48, 25.25s/it]
                                                     
{'loss': 1.4815, 'learning_rate': 0.0002425287356321839, 'epoch': 0.41}

 42%|████▏     | 150/361 [1:02:48<1:28:48, 25.25s/it]
 42%|████▏     | 151/361 [1:03:13<1:28:50, 25.38s/it]
                                                     
{'loss': 1.6877, 'learning_rate': 0.00024137931034482756, 'epoch': 0.42}

 42%|████▏     | 151/361 [1:03:13<1:28:50, 25.38s/it]
 42%|████▏     | 152/361 [1:03:39<1:28:27, 25.40s/it]
                                                     
{'loss': 1.543, 'learning_rate': 0.00024022988505747125, 'epoch': 0.42}

 42%|████▏     | 152/361 [1:03:39<1:28:27, 25.40s/it]
 42%|████▏     | 153/361 [1:04:04<1:28:15, 25.46s/it]
                                                     
{'loss': 1.5718, 'learning_rate': 0.0002390804597701149, 'epoch': 0.42}

 42%|████▏     | 153/361 [1:04:04<1:28:15, 25.46s/it]
 43%|████▎     | 154/361 [1:04:29<1:27:22, 25.32s/it]
                                                     
{'loss': 1.531, 'learning_rate': 0.00023793103448275862, 'epoch': 0.43}

 43%|████▎     | 154/361 [1:04:29<1:27:22, 25.32s/it]
 43%|████▎     | 155/361 [1:04:54<1:26:11, 25.11s/it]
                                                     
{'loss': 1.5009, 'learning_rate': 0.00023678160919540228, 'epoch': 0.43}

 43%|████▎     | 155/361 [1:04:54<1:26:11, 25.11s/it]
 43%|████▎     | 156/361 [1:05:20<1:26:24, 25.29s/it]
                                                     
{'loss': 1.5824, 'learning_rate': 0.00023563218390804593, 'epoch': 0.43}

 43%|████▎     | 156/361 [1:05:20<1:26:24, 25.29s/it]
 43%|████▎     | 157/361 [1:05:43<1:23:50, 24.66s/it]
                                                     
{'loss': 1.5654, 'learning_rate': 0.00023448275862068965, 'epoch': 0.43}

 43%|████▎     | 157/361 [1:05:43<1:23:50, 24.66s/it]
 44%|████▍     | 158/361 [1:06:08<1:23:35, 24.70s/it]
                                                     
{'loss': 1.5385, 'learning_rate': 0.0002333333333333333, 'epoch': 0.44}

 44%|████▍     | 158/361 [1:06:08<1:23:35, 24.70s/it]
 44%|████▍     | 159/361 [1:06:32<1:23:12, 24.72s/it]
                                                     
{'loss': 1.6284, 'learning_rate': 0.000232183908045977, 'epoch': 0.44}

 44%|████▍     | 159/361 [1:06:32<1:23:12, 24.72s/it]
 44%|████▍     | 160/361 [1:06:57<1:22:25, 24.61s/it]
                                                     
{'loss': 1.6007, 'learning_rate': 0.00023103448275862065, 'epoch': 0.44}

 44%|████▍     | 160/361 [1:06:57<1:22:25, 24.61s/it]
 45%|████▍     | 161/361 [1:07:22<1:23:01, 24.91s/it]
                                                     
{'loss': 1.5582, 'learning_rate': 0.00022988505747126436, 'epoch': 0.45}

 45%|████▍     | 161/361 [1:07:22<1:23:01, 24.91s/it]
 45%|████▍     | 162/361 [1:07:48<1:23:23, 25.15s/it]
                                                     
{'loss': 1.5077, 'learning_rate': 0.00022873563218390802, 'epoch': 0.45}

 45%|████▍     | 162/361 [1:07:48<1:23:23, 25.15s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 45%|████▌     | 163/361 [1:08:12<1:21:52, 24.81s/it]
                                                     
{'loss': 1.5516, 'learning_rate': 0.0002275862068965517, 'epoch': 0.45}

 45%|████▌     | 163/361 [1:08:12<1:21:52, 24.81s/it]
 45%|████▌     | 164/361 [1:08:37<1:21:54, 24.95s/it]
                                                     
{'loss': 1.6285, 'learning_rate': 0.0002264367816091954, 'epoch': 0.45}

 45%|████▌     | 164/361 [1:08:37<1:21:54, 24.95s/it]
 46%|████▌     | 165/361 [1:09:03<1:22:14, 25.18s/it]
                                                     
{'loss': 1.5224, 'learning_rate': 0.00022528735632183907, 'epoch': 0.46}

 46%|████▌     | 165/361 [1:09:03<1:22:14, 25.18s/it]
 46%|████▌     | 166/361 [1:09:30<1:23:17, 25.63s/it]
                                                     
{'loss': 1.6383, 'learning_rate': 0.00022413793103448273, 'epoch': 0.46}

 46%|████▌     | 166/361 [1:09:30<1:23:17, 25.63s/it]
 46%|████▋     | 167/361 [1:09:54<1:21:51, 25.32s/it]
                                                     
{'loss': 1.5242, 'learning_rate': 0.00022298850574712644, 'epoch': 0.46}

 46%|████▋     | 167/361 [1:09:54<1:21:51, 25.32s/it]
 47%|████▋     | 168/361 [1:10:19<1:20:33, 25.04s/it]
                                                     
{'loss': 1.5597, 'learning_rate': 0.0002218390804597701, 'epoch': 0.46}

 47%|████▋     | 168/361 [1:10:19<1:20:33, 25.04s/it]
 47%|████▋     | 169/361 [1:10:43<1:19:44, 24.92s/it]
                                                     
{'loss': 1.5981, 'learning_rate': 0.00022068965517241376, 'epoch': 0.47}

 47%|████▋     | 169/361 [1:10:43<1:19:44, 24.92s/it]
 47%|████▋     | 170/361 [1:11:09<1:19:34, 24.99s/it]
                                                     
{'loss': 1.5771, 'learning_rate': 0.00021954022988505744, 'epoch': 0.47}

 47%|████▋     | 170/361 [1:11:09<1:19:34, 24.99s/it]
 47%|████▋     | 171/361 [1:11:33<1:18:48, 24.88s/it]
                                                     
{'loss': 1.5357, 'learning_rate': 0.00021839080459770113, 'epoch': 0.47}

 47%|████▋     | 171/361 [1:11:33<1:18:48, 24.88s/it]
 48%|████▊     | 172/361 [1:11:58<1:18:09, 24.81s/it]
                                                     
{'loss': 1.6067, 'learning_rate': 0.00021724137931034481, 'epoch': 0.48}

 48%|████▊     | 172/361 [1:11:58<1:18:09, 24.81s/it]
 48%|████▊     | 173/361 [1:12:24<1:18:37, 25.09s/it]
                                                     
{'loss': 1.4992, 'learning_rate': 0.00021609195402298847, 'epoch': 0.48}

 48%|████▊     | 173/361 [1:12:24<1:18:37, 25.09s/it]
 48%|████▊     | 174/361 [1:12:49<1:18:18, 25.12s/it]
                                                     
{'loss': 1.5637, 'learning_rate': 0.00021494252873563218, 'epoch': 0.48}

 48%|████▊     | 174/361 [1:12:49<1:18:18, 25.12s/it]
 48%|████▊     | 175/361 [1:13:13<1:17:03, 24.86s/it]
                                                     
{'loss': 1.5477, 'learning_rate': 0.00021379310344827584, 'epoch': 0.48}

 48%|████▊     | 175/361 [1:13:13<1:17:03, 24.86s/it]
 49%|████▉     | 176/361 [1:13:39<1:17:39, 25.19s/it]
                                                     
{'loss': 1.565, 'learning_rate': 0.00021264367816091953, 'epoch': 0.49}

 49%|████▉     | 176/361 [1:13:39<1:17:39, 25.19s/it]
 49%|████▉     | 177/361 [1:14:04<1:17:23, 25.24s/it]
                                                     
{'loss': 1.5481, 'learning_rate': 0.00021149425287356319, 'epoch': 0.49}

 49%|████▉     | 177/361 [1:14:04<1:17:23, 25.24s/it]
 49%|████▉     | 178/361 [1:14:29<1:16:52, 25.20s/it]
                                                     
{'loss': 1.6325, 'learning_rate': 0.0002103448275862069, 'epoch': 0.49}

 49%|████▉     | 178/361 [1:14:29<1:16:52, 25.20s/it]
 50%|████▉     | 179/361 [1:14:54<1:16:12, 25.12s/it]
                                                     
{'loss': 1.646, 'learning_rate': 0.00020919540229885056, 'epoch': 0.49}

 50%|████▉     | 179/361 [1:14:54<1:16:12, 25.12s/it]
 50%|████▉     | 180/361 [1:15:20<1:16:31, 25.37s/it]
                                                     
{'loss': 1.5507, 'learning_rate': 0.00020804597701149421, 'epoch': 0.5}

 50%|████▉     | 180/361 [1:15:20<1:16:31, 25.37s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 50%|█████     | 181/361 [1:15:44<1:14:22, 24.79s/it]
                                                     
{'loss': 1.4867, 'learning_rate': 0.00020689655172413793, 'epoch': 0.5}

 50%|█████     | 181/361 [1:15:44<1:14:22, 24.79s/it]
 50%|█████     | 182/361 [1:16:08<1:13:37, 24.68s/it]
                                                     
{'loss': 1.6861, 'learning_rate': 0.00020574712643678158, 'epoch': 0.5}

 50%|█████     | 182/361 [1:16:08<1:13:37, 24.68s/it]
 51%|█████     | 183/361 [1:16:33<1:13:26, 24.75s/it]
                                                     
{'loss': 1.5232, 'learning_rate': 0.00020459770114942527, 'epoch': 0.51}

 51%|█████     | 183/361 [1:16:33<1:13:26, 24.75s/it]
 51%|█████     | 184/361 [1:16:59<1:13:36, 24.95s/it]
                                                     
{'loss': 1.5858, 'learning_rate': 0.00020344827586206893, 'epoch': 0.51}

 51%|█████     | 184/361 [1:16:59<1:13:36, 24.95s/it]
 51%|█████     | 185/361 [1:17:24<1:13:38, 25.10s/it]
                                                     
{'loss': 1.5106, 'learning_rate': 0.00020229885057471264, 'epoch': 0.51}

 51%|█████     | 185/361 [1:17:24<1:13:38, 25.10s/it]
 52%|█████▏    | 186/361 [1:17:48<1:12:32, 24.87s/it]
                                                     
{'loss': 1.537, 'learning_rate': 0.0002011494252873563, 'epoch': 0.51}

 52%|█████▏    | 186/361 [1:17:48<1:12:32, 24.87s/it]
 52%|█████▏    | 187/361 [1:18:15<1:13:18, 25.28s/it]
                                                     
{'loss': 1.5712, 'learning_rate': 0.00019999999999999998, 'epoch': 0.52}

 52%|█████▏    | 187/361 [1:18:15<1:13:18, 25.28s/it]
 52%|█████▏    | 188/361 [1:18:39<1:12:25, 25.12s/it]
                                                     
{'loss': 1.5099, 'learning_rate': 0.00019885057471264367, 'epoch': 0.52}

 52%|█████▏    | 188/361 [1:18:39<1:12:25, 25.12s/it]
 52%|█████▏    | 189/361 [1:19:04<1:11:42, 25.01s/it]
                                                     
{'loss': 1.6377, 'learning_rate': 0.00019770114942528735, 'epoch': 0.52}

 52%|█████▏    | 189/361 [1:19:04<1:11:42, 25.01s/it]
 53%|█████▎    | 190/361 [1:19:29<1:11:24, 25.06s/it]
                                                     
{'loss': 1.5576, 'learning_rate': 0.000196551724137931, 'epoch': 0.53}

 53%|█████▎    | 190/361 [1:19:29<1:11:24, 25.06s/it]
 53%|█████▎    | 191/361 [1:19:55<1:11:49, 25.35s/it]
                                                     
{'loss': 1.5563, 'learning_rate': 0.00019540229885057467, 'epoch': 0.53}

 53%|█████▎    | 191/361 [1:19:55<1:11:49, 25.35s/it]
 53%|█████▎    | 192/361 [1:20:22<1:12:33, 25.76s/it]
                                                     
{'loss': 1.5521, 'learning_rate': 0.00019425287356321838, 'epoch': 0.53}

 53%|█████▎    | 192/361 [1:20:22<1:12:33, 25.76s/it]
 53%|█████▎    | 193/361 [1:20:48<1:12:07, 25.76s/it]
                                                     
{'loss': 1.5538, 'learning_rate': 0.00019310344827586204, 'epoch': 0.53}

 53%|█████▎    | 193/361 [1:20:48<1:12:07, 25.76s/it]
 54%|█████▎    | 194/361 [1:21:14<1:11:49, 25.81s/it]
                                                     
{'loss': 1.5075, 'learning_rate': 0.00019195402298850572, 'epoch': 0.54}

 54%|█████▎    | 194/361 [1:21:14<1:11:49, 25.81s/it]
 54%|█████▍    | 195/361 [1:21:38<1:10:28, 25.47s/it]
                                                     
{'loss': 1.6379, 'learning_rate': 0.0001908045977011494, 'epoch': 0.54}

 54%|█████▍    | 195/361 [1:21:38<1:10:28, 25.47s/it]
 54%|█████▍    | 196/361 [1:22:04<1:10:25, 25.61s/it]
                                                     
{'loss': 1.6774, 'learning_rate': 0.0001896551724137931, 'epoch': 0.54}

 54%|█████▍    | 196/361 [1:22:04<1:10:25, 25.61s/it]
 55%|█████▍    | 197/361 [1:22:29<1:09:07, 25.29s/it]
                                                     
{'loss': 1.6714, 'learning_rate': 0.00018850574712643675, 'epoch': 0.54}

 55%|█████▍    | 197/361 [1:22:29<1:09:07, 25.29s/it]
 55%|█████▍    | 198/361 [1:22:54<1:08:21, 25.16s/it]
                                                     
{'loss': 1.572, 'learning_rate': 0.00018735632183908046, 'epoch': 0.55}

 55%|█████▍    | 198/361 [1:22:54<1:08:21, 25.16s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 55%|█████▌    | 199/361 [1:23:20<1:08:28, 25.36s/it]
                                                     
{'loss': 1.5978, 'learning_rate': 0.00018620689655172412, 'epoch': 0.55}

 55%|█████▌    | 199/361 [1:23:20<1:08:28, 25.36s/it]
 55%|█████▌    | 200/361 [1:23:44<1:07:24, 25.12s/it]
                                                     
{'loss': 1.6194, 'learning_rate': 0.0001850574712643678, 'epoch': 0.55}

 55%|█████▌    | 200/361 [1:23:44<1:07:24, 25.12s/it]
 56%|█████▌    | 201/361 [1:24:10<1:07:20, 25.25s/it]
                                                     
{'loss': 1.5513, 'learning_rate': 0.00018390804597701147, 'epoch': 0.56}

 56%|█████▌    | 201/361 [1:24:10<1:07:20, 25.25s/it]
 56%|█████▌    | 202/361 [1:24:34<1:06:02, 24.92s/it]
                                                     
{'loss': 1.5873, 'learning_rate': 0.00018275862068965518, 'epoch': 0.56}

 56%|█████▌    | 202/361 [1:24:34<1:06:02, 24.92s/it]
 56%|█████▌    | 203/361 [1:24:59<1:05:56, 25.04s/it]
                                                     
{'loss': 1.5937, 'learning_rate': 0.00018160919540229884, 'epoch': 0.56}

 56%|█████▌    | 203/361 [1:24:59<1:05:56, 25.04s/it]
 57%|█████▋    | 204/361 [1:25:22<1:04:14, 24.55s/it]
                                                     
{'loss': 1.5858, 'learning_rate': 0.0001804597701149425, 'epoch': 0.56}

 57%|█████▋    | 204/361 [1:25:23<1:04:14, 24.55s/it]
 57%|█████▋    | 205/361 [1:25:47<1:03:51, 24.56s/it]
                                                     
{'loss': 1.563, 'learning_rate': 0.0001793103448275862, 'epoch': 0.57}

 57%|█████▋    | 205/361 [1:25:47<1:03:51, 24.56s/it]
 57%|█████▋    | 206/361 [1:26:11<1:03:02, 24.40s/it]
                                                     
{'loss': 1.5386, 'learning_rate': 0.00017816091954022986, 'epoch': 0.57}

 57%|█████▋    | 206/361 [1:26:11<1:03:02, 24.40s/it]
 57%|█████▋    | 207/361 [1:26:36<1:03:05, 24.58s/it]
                                                     
{'loss': 1.595, 'learning_rate': 0.00017701149425287355, 'epoch': 0.57}

 57%|█████▋    | 207/361 [1:26:36<1:03:05, 24.58s/it]
 58%|█████▊    | 208/361 [1:27:02<1:03:45, 25.01s/it]
                                                     
{'loss': 1.5928, 'learning_rate': 0.0001758620689655172, 'epoch': 0.58}

 58%|█████▊    | 208/361 [1:27:02<1:03:45, 25.01s/it]
 58%|█████▊    | 209/361 [1:27:26<1:02:50, 24.81s/it]
                                                     
{'loss': 1.532, 'learning_rate': 0.00017471264367816092, 'epoch': 0.58}

 58%|█████▊    | 209/361 [1:27:26<1:02:50, 24.81s/it]
 58%|█████▊    | 210/361 [1:27:51<1:01:52, 24.59s/it]
                                                     
{'loss': 1.5828, 'learning_rate': 0.00017356321839080458, 'epoch': 0.58}

 58%|█████▊    | 210/361 [1:27:51<1:01:52, 24.59s/it]
 58%|█████▊    | 211/361 [1:28:16<1:02:10, 24.87s/it]
                                                     
{'loss': 1.6477, 'learning_rate': 0.00017241379310344826, 'epoch': 0.58}

 58%|█████▊    | 211/361 [1:28:16<1:02:10, 24.87s/it]
 59%|█████▊    | 212/361 [1:28:42<1:02:18, 25.09s/it]
                                                     
{'loss': 1.5444, 'learning_rate': 0.00017126436781609195, 'epoch': 0.59}

 59%|█████▊    | 212/361 [1:28:42<1:02:18, 25.09s/it]
 59%|█████▉    | 213/361 [1:29:07<1:02:23, 25.29s/it]
                                                     
{'loss': 1.5951, 'learning_rate': 0.00017011494252873563, 'epoch': 0.59}

 59%|█████▉    | 213/361 [1:29:07<1:02:23, 25.29s/it]
 59%|█████▉    | 214/361 [1:29:32<1:01:38, 25.16s/it]
                                                     
{'loss': 1.559, 'learning_rate': 0.0001689655172413793, 'epoch': 0.59}

 59%|█████▉    | 214/361 [1:29:32<1:01:38, 25.16s/it]
 60%|█████▉    | 215/361 [1:29:56<1:00:13, 24.75s/it]
                                                     
{'loss': 1.4873, 'learning_rate': 0.00016781609195402295, 'epoch': 0.59}

 60%|█████▉    | 215/361 [1:29:56<1:00:13, 24.75s/it]
 60%|█████▉    | 216/361 [1:30:21<59:56, 24.81s/it]  
                                                   
{'loss': 1.5619, 'learning_rate': 0.00016666666666666666, 'epoch': 0.6}

 60%|█████▉    | 216/361 [1:30:21<59:56, 24.81s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 60%|██████    | 217/361 [1:30:47<1:00:01, 25.01s/it]
                                                     
{'loss': 1.6061, 'learning_rate': 0.00016551724137931032, 'epoch': 0.6}

 60%|██████    | 217/361 [1:30:47<1:00:01, 25.01s/it]
 60%|██████    | 218/361 [1:31:09<58:09, 24.40s/it]  
                                                   
{'loss': 1.5904, 'learning_rate': 0.000164367816091954, 'epoch': 0.6}

 60%|██████    | 218/361 [1:31:10<58:09, 24.40s/it]
 61%|██████    | 219/361 [1:31:34<57:33, 24.32s/it]
                                                   
{'loss': 1.5271, 'learning_rate': 0.0001632183908045977, 'epoch': 0.61}

 61%|██████    | 219/361 [1:31:34<57:33, 24.32s/it]
 61%|██████    | 220/361 [1:31:59<57:59, 24.67s/it]
                                                   
{'loss': 1.5949, 'learning_rate': 0.00016206896551724137, 'epoch': 0.61}

 61%|██████    | 220/361 [1:31:59<57:59, 24.67s/it]
 61%|██████    | 221/361 [1:32:25<58:30, 25.08s/it]
                                                   
{'loss': 1.4695, 'learning_rate': 0.00016091954022988503, 'epoch': 0.61}

 61%|██████    | 221/361 [1:32:25<58:30, 25.08s/it]
 61%|██████▏   | 222/361 [1:32:51<58:18, 25.17s/it]
                                                   
{'loss': 1.5449, 'learning_rate': 0.00015977011494252874, 'epoch': 0.61}

 61%|██████▏   | 222/361 [1:32:51<58:18, 25.17s/it]
 62%|██████▏   | 223/361 [1:33:15<57:44, 25.10s/it]
                                                   
{'loss': 1.6152, 'learning_rate': 0.0001586206896551724, 'epoch': 0.62}

 62%|██████▏   | 223/361 [1:33:15<57:44, 25.10s/it]
 62%|██████▏   | 224/361 [1:33:42<58:17, 25.53s/it]
                                                   
{'loss': 1.5103, 'learning_rate': 0.0001574712643678161, 'epoch': 0.62}

 62%|██████▏   | 224/361 [1:33:42<58:17, 25.53s/it]
 62%|██████▏   | 225/361 [1:34:07<57:42, 25.46s/it]
                                                   
{'loss': 1.6581, 'learning_rate': 0.00015632183908045975, 'epoch': 0.62}

 62%|██████▏   | 225/361 [1:34:07<57:42, 25.46s/it]
 63%|██████▎   | 226/361 [1:34:32<56:55, 25.30s/it]
                                                   
{'loss': 1.5208, 'learning_rate': 0.00015517241379310346, 'epoch': 0.62}

 63%|██████▎   | 226/361 [1:34:32<56:55, 25.30s/it]
 63%|██████▎   | 227/361 [1:34:58<56:48, 25.44s/it]
                                                   
{'loss': 1.5073, 'learning_rate': 0.00015402298850574712, 'epoch': 0.63}

 63%|██████▎   | 227/361 [1:34:58<56:48, 25.44s/it]
 63%|██████▎   | 228/361 [1:35:23<55:54, 25.23s/it]
                                                   
{'loss': 1.5832, 'learning_rate': 0.00015287356321839077, 'epoch': 0.63}

 63%|██████▎   | 228/361 [1:35:23<55:54, 25.23s/it]
 63%|██████▎   | 229/361 [1:35:48<55:49, 25.37s/it]
                                                   
{'loss': 1.5472, 'learning_rate': 0.00015172413793103449, 'epoch': 0.63}

 63%|██████▎   | 229/361 [1:35:48<55:49, 25.37s/it]
 64%|██████▎   | 230/361 [1:36:15<56:15, 25.77s/it]
                                                   
{'loss': 1.4525, 'learning_rate': 0.00015057471264367814, 'epoch': 0.64}

 64%|██████▎   | 230/361 [1:36:15<56:15, 25.77s/it]
 64%|██████▍   | 231/361 [1:36:40<55:27, 25.59s/it]
                                                   
{'loss': 1.5658, 'learning_rate': 0.00014942528735632183, 'epoch': 0.64}

 64%|██████▍   | 231/361 [1:36:40<55:27, 25.59s/it]
 64%|██████▍   | 232/361 [1:37:05<54:42, 25.45s/it]
                                                   
{'loss': 1.6115, 'learning_rate': 0.00014827586206896551, 'epoch': 0.64}

 64%|██████▍   | 232/361 [1:37:05<54:42, 25.45s/it]
 65%|██████▍   | 233/361 [1:37:30<53:29, 25.08s/it]
                                                   
{'loss': 1.5189, 'learning_rate': 0.00014712643678160917, 'epoch': 0.64}

 65%|██████▍   | 233/361 [1:37:30<53:29, 25.08s/it]
 65%|██████▍   | 234/361 [1:37:54<52:27, 24.78s/it]
                                                   
{'loss': 1.607, 'learning_rate': 0.00014597701149425286, 'epoch': 0.65}

 65%|██████▍   | 234/361 [1:37:54<52:27, 24.78s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 65%|██████▌   | 235/361 [1:38:19<52:33, 25.03s/it]
                                                   
{'loss': 1.6097, 'learning_rate': 0.00014482758620689654, 'epoch': 0.65}

 65%|██████▌   | 235/361 [1:38:19<52:33, 25.03s/it]
 65%|██████▌   | 236/361 [1:38:43<51:04, 24.52s/it]
                                                   
{'loss': 1.593, 'learning_rate': 0.0001436781609195402, 'epoch': 0.65}

 65%|██████▌   | 236/361 [1:38:43<51:04, 24.52s/it]
 66%|██████▌   | 237/361 [1:39:08<51:25, 24.88s/it]
                                                   
{'loss': 1.5749, 'learning_rate': 0.00014252873563218389, 'epoch': 0.66}

 66%|██████▌   | 237/361 [1:39:08<51:25, 24.88s/it]
 66%|██████▌   | 238/361 [1:39:33<50:43, 24.74s/it]
                                                   
{'loss': 1.5324, 'learning_rate': 0.00014137931034482757, 'epoch': 0.66}

 66%|██████▌   | 238/361 [1:39:33<50:43, 24.74s/it]
 66%|██████▌   | 239/361 [1:39:57<50:09, 24.67s/it]
                                                   
{'loss': 1.5747, 'learning_rate': 0.00014022988505747126, 'epoch': 0.66}

 66%|██████▌   | 239/361 [1:39:57<50:09, 24.67s/it]
 66%|██████▋   | 240/361 [1:40:24<50:52, 25.23s/it]
                                                   
{'loss': 1.5534, 'learning_rate': 0.00013908045977011494, 'epoch': 0.66}

 66%|██████▋   | 240/361 [1:40:24<50:52, 25.23s/it]
 67%|██████▋   | 241/361 [1:40:50<51:07, 25.57s/it]
                                                   
{'loss': 1.5633, 'learning_rate': 0.0001379310344827586, 'epoch': 0.67}

 67%|██████▋   | 241/361 [1:40:50<51:07, 25.57s/it]
 67%|██████▋   | 242/361 [1:41:15<50:27, 25.44s/it]
                                                   
{'loss': 1.4998, 'learning_rate': 0.00013678160919540228, 'epoch': 0.67}

 67%|██████▋   | 242/361 [1:41:15<50:27, 25.44s/it]
 67%|██████▋   | 243/361 [1:41:39<49:17, 25.06s/it]
                                                   
{'loss': 1.5424, 'learning_rate': 0.00013563218390804597, 'epoch': 0.67}

 67%|██████▋   | 243/361 [1:41:40<49:17, 25.06s/it]
 68%|██████▊   | 244/361 [1:42:05<49:21, 25.31s/it]
                                                   
{'loss': 1.602, 'learning_rate': 0.00013448275862068965, 'epoch': 0.67}

 68%|██████▊   | 244/361 [1:42:05<49:21, 25.31s/it]
 68%|██████▊   | 245/361 [1:42:31<48:56, 25.31s/it]
                                                   
{'loss': 1.5677, 'learning_rate': 0.0001333333333333333, 'epoch': 0.68}

 68%|██████▊   | 245/361 [1:42:31<48:56, 25.31s/it]
 68%|██████▊   | 246/361 [1:42:54<47:30, 24.78s/it]
                                                   
{'loss': 1.5973, 'learning_rate': 0.000132183908045977, 'epoch': 0.68}

 68%|██████▊   | 246/361 [1:42:54<47:30, 24.78s/it]
 68%|██████▊   | 247/361 [1:43:20<47:38, 25.07s/it]
                                                   
{'loss': 1.5351, 'learning_rate': 0.00013103448275862068, 'epoch': 0.68}

 68%|██████▊   | 247/361 [1:43:20<47:38, 25.07s/it]
 69%|██████▊   | 248/361 [1:43:44<46:24, 24.64s/it]
                                                   
{'loss': 1.534, 'learning_rate': 0.00012988505747126434, 'epoch': 0.69}

 69%|██████▊   | 248/361 [1:43:44<46:24, 24.64s/it]
 69%|██████▉   | 249/361 [1:44:09<46:12, 24.75s/it]
                                                   
{'loss': 1.5374, 'learning_rate': 0.00012873563218390803, 'epoch': 0.69}

 69%|██████▉   | 249/361 [1:44:09<46:12, 24.75s/it]
 69%|██████▉   | 250/361 [1:44:34<46:09, 24.95s/it]
                                                   
{'loss': 1.4865, 'learning_rate': 0.0001275862068965517, 'epoch': 0.69}

 69%|██████▉   | 250/361 [1:44:34<46:09, 24.95s/it]
 70%|██████▉   | 251/361 [1:44:59<45:48, 24.98s/it]
                                                   
{'loss': 1.7316, 'learning_rate': 0.0001264367816091954, 'epoch': 0.69}

 70%|██████▉   | 251/361 [1:44:59<45:48, 24.98s/it]
 70%|██████▉   | 252/361 [1:45:24<45:19, 24.95s/it]
                                                   
{'loss': 1.5201, 'learning_rate': 0.00012528735632183908, 'epoch': 0.7}

 70%|██████▉   | 252/361 [1:45:24<45:19, 24.95s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 70%|███████   | 253/361 [1:45:51<45:52, 25.49s/it]
                                                   
{'loss': 1.4878, 'learning_rate': 0.00012413793103448274, 'epoch': 0.7}

 70%|███████   | 253/361 [1:45:51<45:52, 25.49s/it]
 70%|███████   | 254/361 [1:46:16<45:33, 25.54s/it]
                                                   
{'loss': 1.5604, 'learning_rate': 0.00012298850574712642, 'epoch': 0.7}

 70%|███████   | 254/361 [1:46:16<45:33, 25.54s/it]
 71%|███████   | 255/361 [1:46:42<44:57, 25.45s/it]
                                                   
{'loss': 1.5126, 'learning_rate': 0.0001218390804597701, 'epoch': 0.71}

 71%|███████   | 255/361 [1:46:42<44:57, 25.45s/it]
 71%|███████   | 256/361 [1:47:07<44:40, 25.53s/it]
                                                   
{'loss': 1.5484, 'learning_rate': 0.00012068965517241378, 'epoch': 0.71}

 71%|███████   | 256/361 [1:47:07<44:40, 25.53s/it]
 71%|███████   | 257/361 [1:47:32<43:51, 25.31s/it]
                                                   
{'loss': 1.5308, 'learning_rate': 0.00011954022988505745, 'epoch': 0.71}

 71%|███████   | 257/361 [1:47:32<43:51, 25.31s/it]
 71%|███████▏  | 258/361 [1:47:57<43:13, 25.18s/it]
                                                   
{'loss': 1.5622, 'learning_rate': 0.00011839080459770114, 'epoch': 0.71}

 71%|███████▏  | 258/361 [1:47:57<43:13, 25.18s/it]
 72%|███████▏  | 259/361 [1:48:22<42:33, 25.03s/it]
                                                   
{'loss': 1.5545, 'learning_rate': 0.00011724137931034482, 'epoch': 0.72}

 72%|███████▏  | 259/361 [1:48:22<42:33, 25.03s/it]
 72%|███████▏  | 260/361 [1:48:47<42:05, 25.01s/it]
                                                   
{'loss': 1.663, 'learning_rate': 0.0001160919540229885, 'epoch': 0.72}

 72%|███████▏  | 260/361 [1:48:47<42:05, 25.01s/it]
 72%|███████▏  | 261/361 [1:49:13<42:23, 25.44s/it]
                                                   
{'loss': 1.6411, 'learning_rate': 0.00011494252873563218, 'epoch': 0.72}

 72%|███████▏  | 261/361 [1:49:13<42:23, 25.44s/it]
 73%|███████▎  | 262/361 [1:49:39<42:07, 25.53s/it]
                                                   
{'loss': 1.5234, 'learning_rate': 0.00011379310344827585, 'epoch': 0.72}

 73%|███████▎  | 262/361 [1:49:39<42:07, 25.53s/it]
 73%|███████▎  | 263/361 [1:50:04<41:18, 25.29s/it]
                                                   
{'loss': 1.6244, 'learning_rate': 0.00011264367816091954, 'epoch': 0.73}

 73%|███████▎  | 263/361 [1:50:04<41:18, 25.29s/it]
 73%|███████▎  | 264/361 [1:50:29<40:50, 25.26s/it]
                                                   
{'loss': 1.5449, 'learning_rate': 0.00011149425287356322, 'epoch': 0.73}

 73%|███████▎  | 264/361 [1:50:29<40:50, 25.26s/it]
 73%|███████▎  | 265/361 [1:50:53<40:00, 25.01s/it]
                                                   
{'loss': 1.534, 'learning_rate': 0.00011034482758620688, 'epoch': 0.73}

 73%|███████▎  | 265/361 [1:50:53<40:00, 25.01s/it]
 74%|███████▎  | 266/361 [1:51:20<40:34, 25.63s/it]
                                                   
{'loss': 1.5253, 'learning_rate': 0.00010919540229885056, 'epoch': 0.74}

 74%|███████▎  | 266/361 [1:51:20<40:34, 25.63s/it]
 74%|███████▍  | 267/361 [1:51:44<39:18, 25.09s/it]
                                                   
{'loss': 1.6361, 'learning_rate': 0.00010804597701149424, 'epoch': 0.74}

 74%|███████▍  | 267/361 [1:51:44<39:18, 25.09s/it]
 74%|███████▍  | 268/361 [1:52:10<39:22, 25.41s/it]
                                                   
{'loss': 1.5286, 'learning_rate': 0.00010689655172413792, 'epoch': 0.74}

 74%|███████▍  | 268/361 [1:52:10<39:22, 25.41s/it]
 75%|███████▍  | 269/361 [1:52:35<38:52, 25.35s/it]
                                                   
{'loss': 1.5989, 'learning_rate': 0.00010574712643678159, 'epoch': 0.74}

 75%|███████▍  | 269/361 [1:52:35<38:52, 25.35s/it]
 75%|███████▍  | 270/361 [1:53:02<38:47, 25.58s/it]
                                                   
{'loss': 1.4853, 'learning_rate': 0.00010459770114942528, 'epoch': 0.75}

 75%|███████▍  | 270/361 [1:53:02<38:47, 25.58s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 75%|███████▌  | 271/361 [1:53:27<38:10, 25.45s/it]
                                                   
{'loss': 1.5166, 'learning_rate': 0.00010344827586206896, 'epoch': 0.75}

 75%|███████▌  | 271/361 [1:53:27<38:10, 25.45s/it]
 75%|███████▌  | 272/361 [1:53:52<37:53, 25.54s/it]
                                                   
{'loss': 1.645, 'learning_rate': 0.00010229885057471263, 'epoch': 0.75}

 75%|███████▌  | 272/361 [1:53:52<37:53, 25.54s/it]
 76%|███████▌  | 273/361 [1:54:18<37:19, 25.45s/it]
                                                   
{'loss': 1.5432, 'learning_rate': 0.00010114942528735632, 'epoch': 0.75}

 76%|███████▌  | 273/361 [1:54:18<37:19, 25.45s/it]
 76%|███████▌  | 274/361 [1:54:43<37:02, 25.54s/it]
                                                   
{'loss': 1.4538, 'learning_rate': 9.999999999999999e-05, 'epoch': 0.76}

 76%|███████▌  | 274/361 [1:54:43<37:02, 25.54s/it]
 76%|███████▌  | 275/361 [1:55:08<36:17, 25.32s/it]
                                                   
{'loss': 1.5017, 'learning_rate': 9.885057471264368e-05, 'epoch': 0.76}

 76%|███████▌  | 275/361 [1:55:08<36:17, 25.32s/it]
 76%|███████▋  | 276/361 [1:55:33<35:42, 25.20s/it]
                                                   
{'loss': 1.6282, 'learning_rate': 9.770114942528733e-05, 'epoch': 0.76}

 76%|███████▋  | 276/361 [1:55:33<35:42, 25.20s/it]
 77%|███████▋  | 277/361 [1:55:59<35:24, 25.30s/it]
                                                   
{'loss': 1.5776, 'learning_rate': 9.655172413793102e-05, 'epoch': 0.77}

 77%|███████▋  | 277/361 [1:55:59<35:24, 25.30s/it]
 77%|███████▋  | 278/361 [1:56:24<34:57, 25.27s/it]
                                                   
{'loss': 1.5282, 'learning_rate': 9.54022988505747e-05, 'epoch': 0.77}

 77%|███████▋  | 278/361 [1:56:24<34:57, 25.27s/it]
 77%|███████▋  | 279/361 [1:56:48<33:58, 24.86s/it]
                                                   
{'loss': 1.5555, 'learning_rate': 9.425287356321838e-05, 'epoch': 0.77}

 77%|███████▋  | 279/361 [1:56:48<33:58, 24.86s/it]
 78%|███████▊  | 280/361 [1:57:13<33:43, 24.98s/it]
                                                   
{'loss': 1.5748, 'learning_rate': 9.310344827586206e-05, 'epoch': 0.77}

 78%|███████▊  | 280/361 [1:57:13<33:43, 24.98s/it]
 78%|███████▊  | 281/361 [1:57:41<34:17, 25.72s/it]
                                                   
{'loss': 1.5344, 'learning_rate': 9.195402298850573e-05, 'epoch': 0.78}

 78%|███████▊  | 281/361 [1:57:41<34:17, 25.72s/it]
 78%|███████▊  | 282/361 [1:58:05<33:25, 25.39s/it]
                                                   
{'loss': 1.5968, 'learning_rate': 9.080459770114942e-05, 'epoch': 0.78}

 78%|███████▊  | 282/361 [1:58:05<33:25, 25.39s/it]
 78%|███████▊  | 283/361 [1:58:32<33:26, 25.72s/it]
                                                   
{'loss': 1.6035, 'learning_rate': 8.96551724137931e-05, 'epoch': 0.78}

 78%|███████▊  | 283/361 [1:58:32<33:26, 25.72s/it]
 79%|███████▊  | 284/361 [1:58:57<32:56, 25.67s/it]
                                                   
{'loss': 1.5692, 'learning_rate': 8.850574712643677e-05, 'epoch': 0.79}

 79%|███████▊  | 284/361 [1:58:57<32:56, 25.67s/it]
 79%|███████▉  | 285/361 [1:59:23<32:28, 25.64s/it]
                                                   
{'loss': 1.5076, 'learning_rate': 8.735632183908046e-05, 'epoch': 0.79}

 79%|███████▉  | 285/361 [1:59:23<32:28, 25.64s/it]
 79%|███████▉  | 286/361 [1:59:47<31:20, 25.07s/it]
                                                   
{'loss': 1.6104, 'learning_rate': 8.620689655172413e-05, 'epoch': 0.79}

 79%|███████▉  | 286/361 [1:59:47<31:20, 25.07s/it]
 80%|███████▉  | 287/361 [2:00:12<31:00, 25.14s/it]
                                                   
{'loss': 1.5922, 'learning_rate': 8.505747126436782e-05, 'epoch': 0.79}

 80%|███████▉  | 287/361 [2:00:12<31:00, 25.14s/it]
 80%|███████▉  | 288/361 [2:00:38<30:55, 25.42s/it]
                                                   
{'loss': 1.5804, 'learning_rate': 8.390804597701147e-05, 'epoch': 0.8}

 80%|███████▉  | 288/361 [2:00:38<30:55, 25.42s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 80%|████████  | 289/361 [2:01:02<30:10, 25.15s/it]
                                                   
{'loss': 1.478, 'learning_rate': 8.275862068965516e-05, 'epoch': 0.8}

 80%|████████  | 289/361 [2:01:02<30:10, 25.15s/it]
 80%|████████  | 290/361 [2:01:29<30:19, 25.62s/it]
                                                   
{'loss': 1.4553, 'learning_rate': 8.160919540229884e-05, 'epoch': 0.8}

 80%|████████  | 290/361 [2:01:29<30:19, 25.62s/it]
 81%|████████  | 291/361 [2:01:55<30:02, 25.75s/it]
                                                   
{'loss': 1.5238, 'learning_rate': 8.045977011494252e-05, 'epoch': 0.8}

 81%|████████  | 291/361 [2:01:55<30:02, 25.75s/it]
 81%|████████  | 292/361 [2:02:22<29:55, 26.02s/it]
                                                   
{'loss': 1.5477, 'learning_rate': 7.93103448275862e-05, 'epoch': 0.81}

 81%|████████  | 292/361 [2:02:22<29:55, 26.02s/it]
 81%|████████  | 293/361 [2:02:47<29:15, 25.82s/it]
                                                   
{'loss': 1.6075, 'learning_rate': 7.816091954022987e-05, 'epoch': 0.81}

 81%|████████  | 293/361 [2:02:47<29:15, 25.82s/it]
 81%|████████▏ | 294/361 [2:03:13<28:59, 25.96s/it]
                                                   
{'loss': 1.4747, 'learning_rate': 7.701149425287356e-05, 'epoch': 0.81}

 81%|████████▏ | 294/361 [2:03:13<28:59, 25.96s/it]
 82%|████████▏ | 295/361 [2:03:38<28:08, 25.59s/it]
                                                   
{'loss': 1.6227, 'learning_rate': 7.586206896551724e-05, 'epoch': 0.82}

 82%|████████▏ | 295/361 [2:03:38<28:08, 25.59s/it]
 82%|████████▏ | 296/361 [2:04:04<27:41, 25.55s/it]
                                                   
{'loss': 1.5667, 'learning_rate': 7.471264367816091e-05, 'epoch': 0.82}

 82%|████████▏ | 296/361 [2:04:04<27:41, 25.55s/it]
 82%|████████▏ | 297/361 [2:04:28<26:51, 25.17s/it]
                                                   
{'loss': 1.5686, 'learning_rate': 7.356321839080459e-05, 'epoch': 0.82}

 82%|████████▏ | 297/361 [2:04:28<26:51, 25.17s/it]
 83%|████████▎ | 298/361 [2:04:52<26:12, 24.96s/it]
                                                   
{'loss': 1.5596, 'learning_rate': 7.241379310344827e-05, 'epoch': 0.82}

 83%|████████▎ | 298/361 [2:04:52<26:12, 24.96s/it]
 83%|████████▎ | 299/361 [2:05:17<25:34, 24.75s/it]
                                                   
{'loss': 1.6246, 'learning_rate': 7.126436781609194e-05, 'epoch': 0.83}

 83%|████████▎ | 299/361 [2:05:17<25:34, 24.75s/it]
 83%|████████▎ | 300/361 [2:05:42<25:19, 24.91s/it]
                                                   
{'loss': 1.5836, 'learning_rate': 7.011494252873563e-05, 'epoch': 0.83}

 83%|████████▎ | 300/361 [2:05:42<25:19, 24.91s/it]
 83%|████████▎ | 301/361 [2:06:06<24:47, 24.79s/it]
                                                   
{'loss': 1.4612, 'learning_rate': 6.89655172413793e-05, 'epoch': 0.83}

 83%|████████▎ | 301/361 [2:06:06<24:47, 24.79s/it]
 84%|████████▎ | 302/361 [2:06:31<24:17, 24.70s/it]
                                                   
{'loss': 1.5427, 'learning_rate': 6.781609195402298e-05, 'epoch': 0.84}

 84%|████████▎ | 302/361 [2:06:31<24:17, 24.70s/it]
 84%|████████▍ | 303/361 [2:06:55<23:40, 24.49s/it]
                                                   
{'loss': 1.5246, 'learning_rate': 6.666666666666666e-05, 'epoch': 0.84}

 84%|████████▍ | 303/361 [2:06:55<23:40, 24.49s/it]
 84%|████████▍ | 304/361 [2:07:20<23:18, 24.54s/it]
                                                   
{'loss': 1.5707, 'learning_rate': 6.551724137931034e-05, 'epoch': 0.84}

 84%|████████▍ | 304/361 [2:07:20<23:18, 24.54s/it]
 84%|████████▍ | 305/361 [2:07:45<23:11, 24.85s/it]
                                                   
{'loss': 1.572, 'learning_rate': 6.436781609195401e-05, 'epoch': 0.84}

 84%|████████▍ | 305/361 [2:07:45<23:11, 24.85s/it]
 85%|████████▍ | 306/361 [2:08:11<22:54, 25.00s/it]
                                                   
{'loss': 1.6398, 'learning_rate': 6.32183908045977e-05, 'epoch': 0.85}

 85%|████████▍ | 306/361 [2:08:11<22:54, 25.00s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 85%|████████▌ | 307/361 [2:08:35<22:27, 24.95s/it]
                                                   
{'loss': 1.5573, 'learning_rate': 6.206896551724137e-05, 'epoch': 0.85}

 85%|████████▌ | 307/361 [2:08:35<22:27, 24.95s/it]
 85%|████████▌ | 308/361 [2:08:59<21:44, 24.60s/it]
                                                   
{'loss': 1.4863, 'learning_rate': 6.091954022988505e-05, 'epoch': 0.85}

 85%|████████▌ | 308/361 [2:08:59<21:44, 24.60s/it]
 86%|████████▌ | 309/361 [2:09:24<21:27, 24.75s/it]
                                                   
{'loss': 1.5176, 'learning_rate': 5.9770114942528726e-05, 'epoch': 0.85}

 86%|████████▌ | 309/361 [2:09:24<21:27, 24.75s/it]
 86%|████████▌ | 310/361 [2:09:50<21:21, 25.13s/it]
                                                   
{'loss': 1.4819, 'learning_rate': 5.862068965517241e-05, 'epoch': 0.86}

 86%|████████▌ | 310/361 [2:09:50<21:21, 25.13s/it]
 86%|████████▌ | 311/361 [2:10:16<21:01, 25.24s/it]
                                                   
{'loss': 1.621, 'learning_rate': 5.747126436781609e-05, 'epoch': 0.86}

 86%|████████▌ | 311/361 [2:10:16<21:01, 25.24s/it]
 86%|████████▋ | 312/361 [2:10:41<20:33, 25.18s/it]
                                                   
{'loss': 1.4684, 'learning_rate': 5.632183908045977e-05, 'epoch': 0.86}

 86%|████████▋ | 312/361 [2:10:41<20:33, 25.18s/it]
 87%|████████▋ | 313/361 [2:11:06<20:02, 25.06s/it]
                                                   
{'loss': 1.5143, 'learning_rate': 5.517241379310344e-05, 'epoch': 0.87}

 87%|████████▋ | 313/361 [2:11:06<20:02, 25.06s/it]
 87%|████████▋ | 314/361 [2:11:29<19:21, 24.70s/it]
                                                   
{'loss': 1.5578, 'learning_rate': 5.402298850574712e-05, 'epoch': 0.87}

 87%|████████▋ | 314/361 [2:11:29<19:21, 24.70s/it]
 87%|████████▋ | 315/361 [2:11:56<19:19, 25.20s/it]
                                                   
{'loss': 1.5601, 'learning_rate': 5.2873563218390796e-05, 'epoch': 0.87}

 87%|████████▋ | 315/361 [2:11:56<19:19, 25.20s/it]
 88%|████████▊ | 316/361 [2:12:21<18:47, 25.06s/it]
                                                   
{'loss': 1.5452, 'learning_rate': 5.172413793103448e-05, 'epoch': 0.87}

 88%|████████▊ | 316/361 [2:12:21<18:47, 25.06s/it]
 88%|████████▊ | 317/361 [2:12:45<18:13, 24.86s/it]
                                                   
{'loss': 1.597, 'learning_rate': 5.057471264367816e-05, 'epoch': 0.88}

 88%|████████▊ | 317/361 [2:12:45<18:13, 24.86s/it]
 88%|████████▊ | 318/361 [2:13:10<17:47, 24.82s/it]
                                                   
{'loss': 1.5625, 'learning_rate': 4.942528735632184e-05, 'epoch': 0.88}

 88%|████████▊ | 318/361 [2:13:10<17:47, 24.82s/it]
 88%|████████▊ | 319/361 [2:13:36<17:39, 25.24s/it]
                                                   
{'loss': 1.5702, 'learning_rate': 4.827586206896551e-05, 'epoch': 0.88}

 88%|████████▊ | 319/361 [2:13:36<17:39, 25.24s/it]
 89%|████████▊ | 320/361 [2:14:01<17:07, 25.05s/it]
                                                   
{'loss': 1.532, 'learning_rate': 4.712643678160919e-05, 'epoch': 0.88}

 89%|████████▊ | 320/361 [2:14:01<17:07, 25.05s/it]
 89%|████████▉ | 321/361 [2:14:26<16:42, 25.05s/it]
                                                   
{'loss': 1.5914, 'learning_rate': 4.5977011494252866e-05, 'epoch': 0.89}

 89%|████████▉ | 321/361 [2:14:26<16:42, 25.05s/it]
 89%|████████▉ | 322/361 [2:14:50<16:08, 24.83s/it]
                                                   
{'loss': 1.5147, 'learning_rate': 4.482758620689655e-05, 'epoch': 0.89}

 89%|████████▉ | 322/361 [2:14:50<16:08, 24.83s/it]
 89%|████████▉ | 323/361 [2:15:15<15:45, 24.87s/it]
                                                   
{'loss': 1.5257, 'learning_rate': 4.367816091954023e-05, 'epoch': 0.89}

 89%|████████▉ | 323/361 [2:15:15<15:45, 24.87s/it]
 90%|████████▉ | 324/361 [2:15:41<15:39, 25.40s/it]
                                                   
{'loss': 1.5295, 'learning_rate': 4.252873563218391e-05, 'epoch': 0.9}

 90%|████████▉ | 324/361 [2:15:41<15:39, 25.40s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 90%|█████████ | 325/361 [2:16:08<15:30, 25.84s/it]
                                                   
{'loss': 1.5159, 'learning_rate': 4.137931034482758e-05, 'epoch': 0.9}

 90%|█████████ | 325/361 [2:16:08<15:30, 25.84s/it]
 90%|█████████ | 326/361 [2:16:33<14:55, 25.60s/it]
                                                   
{'loss': 1.5826, 'learning_rate': 4.022988505747126e-05, 'epoch': 0.9}

 90%|█████████ | 326/361 [2:16:33<14:55, 25.60s/it]
 91%|█████████ | 327/361 [2:16:58<14:21, 25.34s/it]
                                                   
{'loss': 1.5299, 'learning_rate': 3.9080459770114936e-05, 'epoch': 0.9}

 91%|█████████ | 327/361 [2:16:58<14:21, 25.34s/it]
 91%|█████████ | 328/361 [2:17:24<14:04, 25.59s/it]
                                                   
{'loss': 1.5306, 'learning_rate': 3.793103448275862e-05, 'epoch': 0.91}

 91%|█████████ | 328/361 [2:17:24<14:04, 25.59s/it]
 91%|█████████ | 329/361 [2:17:51<13:48, 25.88s/it]
                                                   
{'loss': 1.512, 'learning_rate': 3.678160919540229e-05, 'epoch': 0.91}

 91%|█████████ | 329/361 [2:17:51<13:48, 25.88s/it]
 91%|█████████▏| 330/361 [2:18:16<13:13, 25.60s/it]
                                                   
{'loss': 1.565, 'learning_rate': 3.563218390804597e-05, 'epoch': 0.91}

 91%|█████████▏| 330/361 [2:18:16<13:13, 25.60s/it]
 92%|█████████▏| 331/361 [2:18:41<12:41, 25.38s/it]
                                                   
{'loss': 1.5314, 'learning_rate': 3.448275862068965e-05, 'epoch': 0.92}

 92%|█████████▏| 331/361 [2:18:41<12:41, 25.38s/it]
 92%|█████████▏| 332/361 [2:19:07<12:21, 25.56s/it]
                                                   
{'loss': 1.5994, 'learning_rate': 3.333333333333333e-05, 'epoch': 0.92}

 92%|█████████▏| 332/361 [2:19:07<12:21, 25.56s/it]
 92%|█████████▏| 333/361 [2:19:34<12:08, 26.01s/it]
                                                   
{'loss': 1.5003, 'learning_rate': 3.2183908045977006e-05, 'epoch': 0.92}

 92%|█████████▏| 333/361 [2:19:34<12:08, 26.01s/it]
 93%|█████████▎| 334/361 [2:19:59<11:33, 25.69s/it]
                                                   
{'loss': 1.6223, 'learning_rate': 3.1034482758620685e-05, 'epoch': 0.92}

 93%|█████████▎| 334/361 [2:19:59<11:33, 25.69s/it]
 93%|█████████▎| 335/361 [2:20:22<10:49, 25.00s/it]
                                                   
{'loss': 1.5282, 'learning_rate': 2.9885057471264363e-05, 'epoch': 0.93}

 93%|█████████▎| 335/361 [2:20:22<10:49, 25.00s/it]
 93%|█████████▎| 336/361 [2:20:47<10:23, 24.92s/it]
                                                   
{'loss': 1.5282, 'learning_rate': 2.8735632183908045e-05, 'epoch': 0.93}

 93%|█████████▎| 336/361 [2:20:47<10:23, 24.92s/it]
 93%|█████████▎| 337/361 [2:21:11<09:50, 24.58s/it]
                                                   
{'loss': 1.5227, 'learning_rate': 2.758620689655172e-05, 'epoch': 0.93}

 93%|█████████▎| 337/361 [2:21:11<09:50, 24.58s/it]
 94%|█████████▎| 338/361 [2:21:35<09:24, 24.56s/it]
                                                   
{'loss': 1.6007, 'learning_rate': 2.6436781609195398e-05, 'epoch': 0.93}

 94%|█████████▎| 338/361 [2:21:35<09:24, 24.56s/it]
 94%|█████████▍| 339/361 [2:22:00<09:04, 24.76s/it]
                                                   
{'loss': 1.5717, 'learning_rate': 2.528735632183908e-05, 'epoch': 0.94}

 94%|█████████▍| 339/361 [2:22:00<09:04, 24.76s/it]
 94%|█████████▍| 340/361 [2:22:26<08:48, 25.15s/it]
                                                   
{'loss': 1.5208, 'learning_rate': 2.4137931034482755e-05, 'epoch': 0.94}

 94%|█████████▍| 340/361 [2:22:26<08:48, 25.15s/it]
 94%|█████████▍| 341/361 [2:22:52<08:28, 25.41s/it]
                                                   
{'loss': 1.5702, 'learning_rate': 2.2988505747126433e-05, 'epoch': 0.94}

 94%|█████████▍| 341/361 [2:22:52<08:28, 25.41s/it]
 95%|█████████▍| 342/361 [2:23:17<07:59, 25.23s/it]
                                                   
{'loss': 1.564, 'learning_rate': 2.1839080459770115e-05, 'epoch': 0.95}

 95%|█████████▍| 342/361 [2:23:17<07:59, 25.23s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 95%|█████████▌| 343/361 [2:23:43<07:35, 25.32s/it]
                                                   
{'loss': 1.5589, 'learning_rate': 2.068965517241379e-05, 'epoch': 0.95}

 95%|█████████▌| 343/361 [2:23:43<07:35, 25.32s/it]
 95%|█████████▌| 344/361 [2:24:10<07:19, 25.87s/it]
                                                   
{'loss': 1.5234, 'learning_rate': 1.9540229885057468e-05, 'epoch': 0.95}

 95%|█████████▌| 344/361 [2:24:10<07:19, 25.87s/it]
 96%|█████████▌| 345/361 [2:24:35<06:48, 25.51s/it]
                                                   
{'loss': 1.5108, 'learning_rate': 1.8390804597701147e-05, 'epoch': 0.95}

 96%|█████████▌| 345/361 [2:24:35<06:48, 25.51s/it]
 96%|█████████▌| 346/361 [2:24:59<06:18, 25.24s/it]
                                                   
{'loss': 1.5819, 'learning_rate': 1.7241379310344825e-05, 'epoch': 0.96}

 96%|█████████▌| 346/361 [2:24:59<06:18, 25.24s/it]
 96%|█████████▌| 347/361 [2:25:25<05:55, 25.41s/it]
                                                   
{'loss': 1.5429, 'learning_rate': 1.6091954022988503e-05, 'epoch': 0.96}

 96%|█████████▌| 347/361 [2:25:25<05:55, 25.41s/it]
 96%|█████████▋| 348/361 [2:25:49<05:26, 25.10s/it]
                                                   
{'loss': 1.5683, 'learning_rate': 1.4942528735632182e-05, 'epoch': 0.96}

 96%|█████████▋| 348/361 [2:25:49<05:26, 25.10s/it]
 97%|█████████▋| 349/361 [2:26:15<05:03, 25.33s/it]
                                                   
{'loss': 1.5766, 'learning_rate': 1.379310344827586e-05, 'epoch': 0.97}

 97%|█████████▋| 349/361 [2:26:15<05:03, 25.33s/it]
 97%|█████████▋| 350/361 [2:26:40<04:38, 25.30s/it]
                                                   
{'loss': 1.5777, 'learning_rate': 1.264367816091954e-05, 'epoch': 0.97}

 97%|█████████▋| 350/361 [2:26:40<04:38, 25.30s/it]
 97%|█████████▋| 351/361 [2:27:06<04:14, 25.46s/it]
                                                   
{'loss': 1.5347, 'learning_rate': 1.1494252873563217e-05, 'epoch': 0.97}

 97%|█████████▋| 351/361 [2:27:06<04:14, 25.46s/it]
 98%|█████████▊| 352/361 [2:27:33<03:51, 25.75s/it]
                                                   
{'loss': 1.5289, 'learning_rate': 1.0344827586206895e-05, 'epoch': 0.97}

 98%|█████████▊| 352/361 [2:27:33<03:51, 25.75s/it]
 98%|█████████▊| 353/361 [2:27:58<03:24, 25.51s/it]
                                                   
{'loss': 1.552, 'learning_rate': 9.195402298850573e-06, 'epoch': 0.98}

 98%|█████████▊| 353/361 [2:27:58<03:24, 25.51s/it]
 98%|█████████▊| 354/361 [2:28:24<02:59, 25.65s/it]
                                                   
{'loss': 1.4941, 'learning_rate': 8.045977011494252e-06, 'epoch': 0.98}

 98%|█████████▊| 354/361 [2:28:24<02:59, 25.65s/it]
 98%|█████████▊| 355/361 [2:28:47<02:30, 25.06s/it]
                                                   
{'loss': 1.657, 'learning_rate': 6.89655172413793e-06, 'epoch': 0.98}

 98%|█████████▊| 355/361 [2:28:47<02:30, 25.06s/it]
 99%|█████████▊| 356/361 [2:29:14<02:08, 25.61s/it]
                                                   
{'loss': 1.5671, 'learning_rate': 5.747126436781608e-06, 'epoch': 0.98}

 99%|█████████▊| 356/361 [2:29:14<02:08, 25.61s/it]
 99%|█████████▉| 357/361 [2:29:37<01:38, 24.70s/it]
                                                   
{'loss': 1.5094, 'learning_rate': 4.597701149425287e-06, 'epoch': 0.99}

 99%|█████████▉| 357/361 [2:29:37<01:38, 24.70s/it]
 99%|█████████▉| 358/361 [2:30:02<01:14, 24.77s/it]
                                                   
{'loss': 1.4737, 'learning_rate': 3.448275862068965e-06, 'epoch': 0.99}

 99%|█████████▉| 358/361 [2:30:02<01:14, 24.77s/it]
 99%|█████████▉| 359/361 [2:30:28<00:50, 25.21s/it]
                                                   
{'loss': 1.5532, 'learning_rate': 2.2988505747126433e-06, 'epoch': 0.99}

 99%|█████████▉| 359/361 [2:30:28<00:50, 25.21s/it]
100%|█████████▉| 360/361 [2:30:53<00:25, 25.11s/it]
                                                   
{'loss': 1.5128, 'learning_rate': 1.1494252873563217e-06, 'epoch': 1.0}

100%|█████████▉| 360/361 [2:30:53<00:25, 25.11s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

100%|██████████| 361/361 [2:31:18<00:00, 25.06s/it]
                                                   
{'loss': 1.5317, 'learning_rate': 0.0, 'epoch': 1.0}

100%|██████████| 361/361 [2:31:18<00:00, 25.06s/it]
                                                   
{'train_runtime': 9096.6729, 'train_samples_per_second': 5.088, 'train_steps_per_second': 0.04, 'train_loss': 1.60190208169562, 'epoch': 1.0}

100%|██████████| 361/361 [2:31:18<00:00, 25.06s/it]
100%|██████████| 361/361 [2:31:18<00:00, 25.15s/it]

 If there's a warning about missing keys above, please disregard :)
