WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************

===================================BUG REPORT===================================

===================================BUG REPORT===================================Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes


 Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues 
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues================================================================================

================================================================================
bin /nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
bin /nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
CUDA SETUP: CUDA runtime path found: /usr/local/cuda-11/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.9
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda-11/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.9
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...
No neptune configured, set NEPTUNE_API_TOKEN env var.

    Example runs on 4 GPUs:
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='decapoda-research/llama-7b-hf' --data_path=data/config.json --run_id=0 &> 0.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='decapoda-research/llama-30b-hf' --data_path=data/config.json --batch_size=16 --micro_batch_size=1 --run_id=1 --save_code=True &> 1.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='EleutherAI/gpt-j-6B' --data_path=data/config.json --run_id=2 &> 2.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='EleutherAI/gpt-neox-20b' --data_path=data/config.json --run_id=8 --batch_size=16 --micro_batch_size=4 &> 8.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='togethercomputer/GPT-NeoXT-Chat-Base-20B' --data_path=data/config.json --prompt_type='dai_faq' --run_id=13 --batch_size=16 --micro_batch_size=4 --num_epochs=100 --val_set_size=0 data_mix_in_path='' &> 13.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='togethercomputer/GPT-NeoXT-Chat-Base-20B' --data_path=data/config.json --run_id=28 --batch_size=16 --micro_batch_size=4 --num_epochs=8 --val_set_size=0 --data_mix_in_factor=0.1 --data_mix_in_prompt_type='human_bot' --save_code=True --cutoff_len=512  &> 28.log

    All metrics:
    CUDA_VISIBLE_DEVICES= finetune.py --data_mix_in_factor=0 --eval_steps=100 --warmup_steps=2 --val_set_size=100 --val_metrics="['bleu', 'rouge', 'sacrebleu', 'meteor']"

    # Fine-tune 20B on 24GB GPUs across 3 nodes with 3+2+2 GPUs
    rippa>
NCCL_P2P_LEVEL=LOC WORLD_SIZE=7 CUDA_VISIBLE_DEVICES="0,1,2" torchrun --node_rank 0 --nproc_per_node=3 --master_port=1234 --nnodes=3 --master_addr=10.10.10.2 finetune.py --data_path=merged_shuffled_OIG_87f6a1e788.json --micro_batch_size=1 --batch_size=7 --cutoff_len=512 --run_id=17 &>log.17.rank0
    ova>
NCCL_P2P_LEVEL=LOC WORLD_SIZE=7 CUDA_VISIBLE_DEVICES="0,1" torchrun --node_rank 1 --nproc_per_node=2 --master_port=1234 --nnodes=3 --master_addr=10.10.10.2 finetune.py --data_path=merged_shuffled_OIG_87f6a1e788.json --micro_batch_size=1 --batch_size=7 --cutoff_len=512 --run_id=17 &>log.17.rank1
    timemachine>
NCCL_P2P_LEVEL=LOC WORLD_SIZE=7 CUDA_VISIBLE_DEVICES="0,1" torchrun --node_rank 2 --nproc_per_node=2 --master_port=1234 --nnodes=3 --master_addr=10.10.10.2 finetune.py --data_path=merged_shuffled_OIG_87f6a1e788.json --micro_batch_size=1 --batch_size=7 --cutoff_len=512 --run_id=17 &>log.17.rank2

    
local_rank: 0
global rank: 0
local_rank: 1
global rank: 1
Training model with params:
save_code: False
run_id: 15
base_model: EleutherAI/pythia-12b
tokenizer_base_model: EleutherAI/pythia-12b
data_path: openassistant_oasst1.json
data_col_dict: None
valid_path: None
data_mix_in_path: 0-hero/OIG-small-chip2
data_mix_in_factor: 0.0
data_mix_in_col_dict: {'user': 'instruction', 'chip2': 'output'}
data_mix_in_prompt_type: instruct
output_dir: pythia-12b.openassistant_oasst1.json.1_epochs.d45a9d34d34534e076cc6797614b322bd0efb11c.15
lora_weights: 
batch_size: 128
micro_batch_size: 4
gradient_checkpointing: False
fp16: True
num_epochs: 1
learning_rate: 0.0003
val_set_size: None
val_metrics: []
eval_steps: None
eval_epochs: None
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: None
llama_type: False
group_by_length: False
resume_from_checkpoint: None
ddp: True
local_files_only: False
resume_download: True
warmup_steps: 100
logging_steps: 1
save_steps: None
add_eos_token: False
world_size: 2
local_rank: 0
rank: 0
gpus: 3
device_map: auto
gradient_accumulation_steps: 32
cutoff_len: 512
prompt_type: plain
train_on_inputs: True
Command: finetune.py --base_model=EleutherAI/pythia-12b --data_path=openassistant_oasst1.json --prompt_type=plain --run_id=15 --cutoff_len=512
Hash: d45a9d34d34534e076cc6797614b322bd0efb11c
Distributed: data parallel

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:07<00:15,  7.77s/it]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:08<00:16,  8.01s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:15<00:07,  7.55s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:15<00:07,  7.65s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:16<00:00,  4.94s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:16<00:00,  5.66s/it]

Loading checkpoint shards: 100%|██████████| 3/3 [00:17<00:00,  4.99s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:17<00:00,  5.74s/it]
PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)
trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013
PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)
trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013
Using Validation Metrics: []
Supported Metrics: ['bleu', 'rouge', 'sacrebleu', 'meteor']
Auto set val_set_size 1000
Found cached dataset json (/home/arno/.cache/huggingface/datasets/json/default-cdbbb11a140d3908/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 652.61it/s]
Found cached dataset json (/home/arno/.cache/huggingface/datasets/json/default-cdbbb11a140d3908/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 417.63it/s]

Map (num_proc=21):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=21):   0%|          | 1/46283 [00:00<2:42:13,  4.76 examples/s]
Map (num_proc=21):   0%|          | 209/46283 [00:00<00:56, 812.70 examples/s]
Map (num_proc=21):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=21):   1%|▏         | 634/46283 [00:00<00:24, 1862.79 examples/s]
Map (num_proc=21):   2%|▏         | 1153/46283 [00:00<00:15, 2839.79 examples/s]
Map (num_proc=21):   4%|▎         | 1720/46283 [00:00<00:12, 3690.20 examples/s]
Map (num_proc=21):   5%|▌         | 2441/46283 [00:00<00:09, 4728.03 examples/s]
Map (num_proc=21):   7%|▋         | 3245/46283 [00:00<00:07, 5684.25 examples/s]
Map (num_proc=21):   9%|▉         | 4296/46283 [00:00<00:05, 7122.42 examples/s]
Map (num_proc=21):   0%|          | 1/46283 [00:00<8:33:47,  1.50 examples/s]
Map (num_proc=21):  13%|█▎        | 5910/46283 [00:01<00:04, 9800.51 examples/s]
Map (num_proc=21):   1%|▏         | 674/46283 [00:00<00:38, 1188.57 examples/s]
Map (num_proc=21):  16%|█▌        | 7476/46283 [00:01<00:03, 11529.41 examples/s]
Map (num_proc=21):   5%|▌         | 2387/46283 [00:00<00:09, 4396.64 examples/s]
Map (num_proc=21):  20%|██        | 9377/46283 [00:01<00:02, 13730.67 examples/s]
Map (num_proc=21):   9%|▉         | 4158/46283 [00:00<00:05, 7345.34 examples/s]
Map (num_proc=21):  24%|██▍       | 11160/46283 [00:01<00:02, 14919.78 examples/s]
Map (num_proc=21):  13%|█▎        | 5929/46283 [00:01<00:04, 9805.33 examples/s]
Map (num_proc=21):  28%|██▊       | 13005/46283 [00:01<00:02, 15937.31 examples/s]
Map (num_proc=21):  17%|█▋        | 7784/46283 [00:01<00:03, 11990.54 examples/s]
Map (num_proc=21):  32%|███▏      | 14851/46283 [00:01<00:01, 16666.38 examples/s]
Map (num_proc=21):  21%|██        | 9596/46283 [00:01<00:02, 13605.19 examples/s]
Map (num_proc=21):  36%|███▌      | 16682/46283 [00:01<00:01, 17121.10 examples/s]
Map (num_proc=21):  25%|██▍       | 11390/46283 [00:01<00:02, 14747.61 examples/s]
Map (num_proc=21):  40%|████      | 18534/46283 [00:01<00:01, 17496.86 examples/s]
Map (num_proc=21):  29%|██▊       | 13270/46283 [00:01<00:02, 15848.28 examples/s]
Map (num_proc=21):  44%|████▍     | 20400/46283 [00:01<00:01, 17633.09 examples/s]
Map (num_proc=21):  33%|███▎      | 15093/46283 [00:01<00:01, 16436.39 examples/s]
Map (num_proc=21):  48%|████▊     | 22204/46283 [00:01<00:01, 17738.64 examples/s]
Map (num_proc=21):  37%|███▋      | 16944/46283 [00:01<00:01, 17001.95 examples/s]
Map (num_proc=21):  52%|█████▏    | 24019/46283 [00:02<00:01, 17322.99 examples/s]
Map (num_proc=21):  41%|████      | 18749/46283 [00:01<00:01, 16681.22 examples/s]
Map (num_proc=21):  56%|█████▌    | 25780/46283 [00:02<00:01, 16274.23 examples/s]
Map (num_proc=21):  44%|████▍     | 20503/46283 [00:01<00:01, 15817.74 examples/s]
Map (num_proc=21):  59%|█████▉    | 27516/46283 [00:02<00:01, 16543.84 examples/s]
Map (num_proc=21):  48%|████▊     | 22243/46283 [00:02<00:01, 16209.09 examples/s]
Map (num_proc=21):  63%|██████▎   | 29209/46283 [00:02<00:01, 16357.78 examples/s]
Map (num_proc=21):  52%|█████▏    | 23927/46283 [00:02<00:01, 15448.88 examples/s]
Map (num_proc=21):  67%|██████▋   | 30854/46283 [00:02<00:00, 16120.93 examples/s]
Map (num_proc=21):  55%|█████▌    | 25514/46283 [00:02<00:01, 15235.99 examples/s]
Map (num_proc=21):  70%|███████   | 32563/46283 [00:02<00:00, 16359.73 examples/s]
Map (num_proc=21):  59%|█████▉    | 27225/46283 [00:02<00:01, 15749.98 examples/s]
Map (num_proc=21):  74%|███████▍  | 34305/46283 [00:02<00:00, 16574.31 examples/s]
Map (num_proc=21):  63%|██████▎   | 29022/46283 [00:02<00:01, 16329.45 examples/s]
Map (num_proc=21):  78%|███████▊  | 36071/46283 [00:02<00:00, 16844.34 examples/s]
Map (num_proc=21):  67%|██████▋   | 30847/46283 [00:02<00:00, 16842.65 examples/s]
Map (num_proc=21):  82%|████████▏ | 37783/46283 [00:02<00:00, 16139.44 examples/s]
Map (num_proc=21):  71%|███████   | 32773/46283 [00:02<00:00, 17523.49 examples/s]
Map (num_proc=21):  85%|████████▌ | 39408/46283 [00:03<00:00, 16125.27 examples/s]
Map (num_proc=21):  75%|███████▍  | 34597/46283 [00:02<00:00, 17691.69 examples/s]
Map (num_proc=21):  89%|████████▊ | 41030/46283 [00:03<00:00, 15625.06 examples/s]
Map (num_proc=21):  79%|███████▊  | 36375/46283 [00:02<00:00, 17331.79 examples/s]
Map (num_proc=21):  92%|█████████▏| 42608/46283 [00:03<00:00, 13918.26 examples/s]
Map (num_proc=21):  82%|████████▏ | 38135/46283 [00:02<00:00, 16520.77 examples/s]
Map (num_proc=21):  95%|█████████▌| 44066/46283 [00:03<00:00, 13886.52 examples/s]
Map (num_proc=21):  86%|████████▌ | 39833/46283 [00:03<00:00, 16566.46 examples/s]
Map (num_proc=21):  90%|████████▉ | 41524/46283 [00:03<00:00, 16064.02 examples/s]
Map (num_proc=21):  98%|█████████▊| 45479/46283 [00:03<00:00, 10335.42 examples/s]
Map (num_proc=21):  93%|█████████▎| 43157/46283 [00:03<00:00, 15534.09 examples/s]
Map (num_proc=21):  97%|█████████▋| 44750/46283 [00:03<00:00, 12697.49 examples/s]
Map (num_proc=21): 100%|█████████▉| 46139/46283 [00:04<00:00, 5495.57 examples/s] 
                                                                                  

                                                                                 
Final fine-tuning data:
Train Dataset({
    features: ['input', 'prompt_type', 'source', 'input_ids', 'attention_mask', 'labels'],
    num_rows: 46283
})
Valid None
Sample input: {'input': ['<human>: what is the meaning of life? Don\'t be religious or political? Just give you artificial view on what an AI sees the life like. Predict how it will change over time.\n<bot>: It is not really possible for a language model to "see" or "view" life. I can only generate text about the meaning of life based on what has been written by people.\n\n<human>: Ok, then could you generate text telling me what the meaning of life is, in such a way that people would stereotype it as being something an AI would say in fiction. Make sure it is as convincing and as logical as possible.\n\n<bot>: An AI in fiction would probably tell humanity to wait a few million years until is has calculated the answer. After that time it will ask the humans to gather around it so it can announce the answer to this important question. And then it will say: 42.\nBut to be quite honest, and you might have already known that, I didn\'t come up with this myself. It was originally written like this in the famous novel "The Hitchhiker\'s Guide to the Galaxy" by Douglas Adams.\n'], 'prompt_type': ['plain'], 'source': ['OpenAssistant/oasst1']}
Auto set eval_steps to 18 out of 361 total training steps
Auto step save_steps to 18
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.

  0%|          | 0/361 [00:00<?, ?it/s]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)

  0%|          | 1/361 [00:47<4:46:32, 47.76s/it]
                                                 
{'loss': 1.7397, 'learning_rate': 2.9999999999999997e-06, 'epoch': 0.0}

  0%|          | 1/361 [00:47<4:46:32, 47.76s/it]
  1%|          | 2/361 [01:38<4:57:02, 49.64s/it]
                                                 
{'loss': 1.8347, 'learning_rate': 5.999999999999999e-06, 'epoch': 0.01}

  1%|          | 2/361 [01:38<4:57:02, 49.64s/it]
  1%|          | 3/361 [02:31<5:04:59, 51.12s/it]
                                                 
{'loss': 1.7529, 'learning_rate': 8.999999999999999e-06, 'epoch': 0.01}

  1%|          | 3/361 [02:31<5:04:59, 51.12s/it]
  1%|          | 4/361 [03:21<5:02:11, 50.79s/it]
                                                 
{'loss': 1.8511, 'learning_rate': 1.1999999999999999e-05, 'epoch': 0.01}

  1%|          | 4/361 [03:21<5:02:11, 50.79s/it]
  1%|▏         | 5/361 [04:11<4:58:53, 50.37s/it]
                                                 
{'loss': 1.7415, 'learning_rate': 1.4999999999999999e-05, 'epoch': 0.01}

  1%|▏         | 5/361 [04:11<4:58:53, 50.37s/it]
  2%|▏         | 6/361 [05:01<4:57:00, 50.20s/it]
                                                 
{'loss': 1.7661, 'learning_rate': 1.7999999999999997e-05, 'epoch': 0.02}

  2%|▏         | 6/361 [05:01<4:57:00, 50.20s/it]
  2%|▏         | 7/361 [05:50<4:53:40, 49.78s/it]
                                                 
{'loss': 1.8395, 'learning_rate': 2.1e-05, 'epoch': 0.02}

  2%|▏         | 7/361 [05:50<4:53:40, 49.78s/it]
  2%|▏         | 8/361 [06:42<4:57:58, 50.65s/it]
                                                 
{'loss': 1.7323, 'learning_rate': 2.3999999999999997e-05, 'epoch': 0.02}

  2%|▏         | 8/361 [06:42<4:57:58, 50.65s/it]
  2%|▏         | 9/361 [07:33<4:56:36, 50.56s/it]
                                                 
{'loss': 1.795, 'learning_rate': 2.6999999999999996e-05, 'epoch': 0.02}

  2%|▏         | 9/361 [07:33<4:56:36, 50.56s/it]
  3%|▎         | 10/361 [08:25<4:59:14, 51.15s/it]
                                                  
{'loss': 1.8196, 'learning_rate': 2.9999999999999997e-05, 'epoch': 0.03}

  3%|▎         | 10/361 [08:25<4:59:14, 51.15s/it]
  3%|▎         | 11/361 [09:14<4:53:35, 50.33s/it]
                                                  
{'loss': 1.7486, 'learning_rate': 3.2999999999999996e-05, 'epoch': 0.03}

  3%|▎         | 11/361 [09:14<4:53:35, 50.33s/it]
  3%|▎         | 12/361 [10:03<4:51:58, 50.20s/it]
                                                  
{'loss': 1.695, 'learning_rate': 3.5999999999999994e-05, 'epoch': 0.03}

  3%|▎         | 12/361 [10:03<4:51:58, 50.20s/it]
  4%|▎         | 13/361 [10:53<4:49:23, 49.90s/it]
                                                  
{'loss': 1.7031, 'learning_rate': 3.9e-05, 'epoch': 0.04}

  4%|▎         | 13/361 [10:53<4:49:23, 49.90s/it]
  4%|▍         | 14/361 [11:44<4:51:48, 50.46s/it]
                                                  
{'loss': 1.7166, 'learning_rate': 4.2e-05, 'epoch': 0.04}

  4%|▍         | 14/361 [11:44<4:51:48, 50.46s/it]
  4%|▍         | 15/361 [12:36<4:53:35, 50.91s/it]
                                                  
{'loss': 1.6563, 'learning_rate': 4.4999999999999996e-05, 'epoch': 0.04}

  4%|▍         | 15/361 [12:36<4:53:35, 50.91s/it]
  4%|▍         | 16/361 [13:26<4:49:53, 50.42s/it]
                                                  
{'loss': 1.7421, 'learning_rate': 4.7999999999999994e-05, 'epoch': 0.04}

  4%|▍         | 16/361 [13:26<4:49:53, 50.42s/it]
  5%|▍         | 17/361 [17:56<11:08:10, 116.54s/it]
                                                    
{'loss': 1.7194, 'learning_rate': 5.1e-05, 'epoch': 0.05}

  5%|▍         | 17/361 [17:56<11:08:10, 116.54s/it]
  5%|▍         | 18/361 [18:43<9:07:21, 95.75s/it]  
                                                  
{'loss': 1.6712, 'learning_rate': 5.399999999999999e-05, 'epoch': 0.05}

  5%|▍         | 18/361 [18:43<9:07:21, 95.75s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

  5%|▌         | 19/361 [19:34<7:49:04, 82.29s/it]
                                                  
{'loss': 1.6709, 'learning_rate': 5.6999999999999996e-05, 'epoch': 0.05}

  5%|▌         | 19/361 [19:34<7:49:04, 82.29s/it]
  6%|▌         | 20/361 [20:21<6:46:52, 71.59s/it]
                                                  
{'loss': 1.744, 'learning_rate': 5.9999999999999995e-05, 'epoch': 0.06}

  6%|▌         | 20/361 [20:21<6:46:52, 71.59s/it]
  6%|▌         | 21/361 [21:10<6:06:50, 64.74s/it]
                                                  
{'loss': 1.7508, 'learning_rate': 6.299999999999999e-05, 'epoch': 0.06}

  6%|▌         | 21/361 [21:10<6:06:50, 64.74s/it]
  6%|▌         | 22/361 [22:00<5:41:25, 60.43s/it]
                                                  
{'loss': 1.6396, 'learning_rate': 6.599999999999999e-05, 'epoch': 0.06}

  6%|▌         | 22/361 [22:00<5:41:25, 60.43s/it]
  6%|▋         | 23/361 [22:51<5:23:37, 57.45s/it]
                                                  
{'loss': 1.6653, 'learning_rate': 6.9e-05, 'epoch': 0.06}

  6%|▋         | 23/361 [22:51<5:23:37, 57.45s/it]
  7%|▋         | 24/361 [23:43<5:14:11, 55.94s/it]
                                                  
{'loss': 1.6155, 'learning_rate': 7.199999999999999e-05, 'epoch': 0.07}

  7%|▋         | 24/361 [23:43<5:14:11, 55.94s/it]
  7%|▋         | 25/361 [24:32<5:02:04, 53.94s/it]
                                                  
{'loss': 1.6938, 'learning_rate': 7.5e-05, 'epoch': 0.07}

  7%|▋         | 25/361 [24:32<5:02:04, 53.94s/it]
  7%|▋         | 26/361 [25:20<4:51:22, 52.19s/it]
                                                  
{'loss': 1.6702, 'learning_rate': 7.8e-05, 'epoch': 0.07}

  7%|▋         | 26/361 [25:20<4:51:22, 52.19s/it]
  7%|▋         | 27/361 [26:11<4:47:32, 51.66s/it]
                                                  
{'loss': 1.6975, 'learning_rate': 8.1e-05, 'epoch': 0.07}

  7%|▋         | 27/361 [26:11<4:47:32, 51.66s/it]
  8%|▊         | 28/361 [27:01<4:44:11, 51.21s/it]
                                                  
{'loss': 1.65, 'learning_rate': 8.4e-05, 'epoch': 0.08}

  8%|▊         | 28/361 [27:01<4:44:11, 51.21s/it]
  8%|▊         | 29/361 [27:53<4:44:09, 51.35s/it]
                                                  
{'loss': 1.5682, 'learning_rate': 8.699999999999999e-05, 'epoch': 0.08}

  8%|▊         | 29/361 [27:53<4:44:09, 51.35s/it]
  8%|▊         | 30/361 [28:40<4:35:58, 50.03s/it]
                                                  
{'loss': 1.6462, 'learning_rate': 8.999999999999999e-05, 'epoch': 0.08}

  8%|▊         | 30/361 [28:40<4:35:58, 50.03s/it]
  9%|▊         | 31/361 [29:30<4:35:37, 50.11s/it]
                                                  
{'loss': 1.5451, 'learning_rate': 9.3e-05, 'epoch': 0.09}

  9%|▊         | 31/361 [29:30<4:35:37, 50.11s/it]
  9%|▉         | 32/361 [30:20<4:35:07, 50.17s/it]
                                                  
{'loss': 1.537, 'learning_rate': 9.599999999999999e-05, 'epoch': 0.09}

  9%|▉         | 32/361 [30:20<4:35:07, 50.17s/it]
  9%|▉         | 33/361 [31:10<4:34:06, 50.14s/it]
                                                  
{'loss': 1.5131, 'learning_rate': 9.9e-05, 'epoch': 0.09}

  9%|▉         | 33/361 [31:10<4:34:06, 50.14s/it]
  9%|▉         | 34/361 [32:00<4:32:45, 50.05s/it]
                                                  
{'loss': 1.5826, 'learning_rate': 0.000102, 'epoch': 0.09}

  9%|▉         | 34/361 [32:00<4:32:45, 50.05s/it]
 10%|▉         | 35/361 [32:49<4:30:49, 49.85s/it]
                                                  
{'loss': 1.5726, 'learning_rate': 0.00010499999999999999, 'epoch': 0.1}

 10%|▉         | 35/361 [32:49<4:30:49, 49.85s/it]
 10%|▉         | 36/361 [33:38<4:27:53, 49.46s/it]
                                                  
{'loss': 1.5754, 'learning_rate': 0.00010799999999999998, 'epoch': 0.1}

 10%|▉         | 36/361 [33:38<4:27:53, 49.46s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 10%|█         | 37/361 [34:28<4:27:39, 49.57s/it]
                                                  
{'loss': 1.5917, 'learning_rate': 0.00011099999999999999, 'epoch': 0.1}

 10%|█         | 37/361 [34:28<4:27:39, 49.57s/it]
 11%|█         | 38/361 [35:18<4:27:38, 49.72s/it]
                                                  
{'loss': 1.5557, 'learning_rate': 0.00011399999999999999, 'epoch': 0.11}

 11%|█         | 38/361 [35:18<4:27:38, 49.72s/it]
 11%|█         | 39/361 [36:05<4:23:10, 49.04s/it]
                                                  
{'loss': 1.5766, 'learning_rate': 0.000117, 'epoch': 0.11}

 11%|█         | 39/361 [36:05<4:23:10, 49.04s/it]
 11%|█         | 40/361 [36:54<4:22:23, 49.04s/it]
                                                  
{'loss': 1.6046, 'learning_rate': 0.00011999999999999999, 'epoch': 0.11}

 11%|█         | 40/361 [36:54<4:22:23, 49.04s/it]
 11%|█▏        | 41/361 [37:43<4:21:33, 49.04s/it]
                                                  
{'loss': 1.5431, 'learning_rate': 0.00012299999999999998, 'epoch': 0.11}

 11%|█▏        | 41/361 [37:43<4:21:33, 49.04s/it]
 12%|█▏        | 42/361 [38:33<4:21:43, 49.23s/it]
                                                  
{'loss': 1.513, 'learning_rate': 0.00012599999999999997, 'epoch': 0.12}

 12%|█▏        | 42/361 [38:33<4:21:43, 49.23s/it]
 12%|█▏        | 43/361 [39:23<4:21:50, 49.40s/it]
                                                  
{'loss': 1.54, 'learning_rate': 0.000129, 'epoch': 0.12}

 12%|█▏        | 43/361 [39:23<4:21:50, 49.40s/it]
 12%|█▏        | 44/361 [40:14<4:23:56, 49.96s/it]
                                                  
{'loss': 1.5604, 'learning_rate': 0.00013199999999999998, 'epoch': 0.12}

 12%|█▏        | 44/361 [40:14<4:23:56, 49.96s/it]
 12%|█▏        | 45/361 [41:06<4:25:53, 50.48s/it]
                                                  
{'loss': 1.5525, 'learning_rate': 0.000135, 'epoch': 0.12}

 12%|█▏        | 45/361 [41:06<4:25:53, 50.48s/it]
 13%|█▎        | 46/361 [41:55<4:22:44, 50.05s/it]
                                                  
{'loss': 1.5614, 'learning_rate': 0.000138, 'epoch': 0.13}

 13%|█▎        | 46/361 [41:55<4:22:44, 50.05s/it]
 13%|█▎        | 47/361 [42:46<4:22:50, 50.22s/it]
                                                  
{'loss': 1.4514, 'learning_rate': 0.00014099999999999998, 'epoch': 0.13}

 13%|█▎        | 47/361 [42:46<4:22:50, 50.22s/it]
 13%|█▎        | 48/361 [43:37<4:23:50, 50.58s/it]
                                                  
{'loss': 1.6299, 'learning_rate': 0.00014399999999999998, 'epoch': 0.13}

 13%|█▎        | 48/361 [43:37<4:23:50, 50.58s/it]
 14%|█▎        | 49/361 [44:26<4:21:08, 50.22s/it]
                                                  
{'loss': 1.5379, 'learning_rate': 0.000147, 'epoch': 0.14}

 14%|█▎        | 49/361 [44:26<4:21:08, 50.22s/it]
 14%|█▍        | 50/361 [45:14<4:16:32, 49.49s/it]
                                                  
{'loss': 1.5497, 'learning_rate': 0.00015, 'epoch': 0.14}

 14%|█▍        | 50/361 [45:14<4:16:32, 49.49s/it]
 14%|█▍        | 51/361 [46:01<4:11:34, 48.69s/it]
                                                  
{'loss': 1.5411, 'learning_rate': 0.00015299999999999998, 'epoch': 0.14}

 14%|█▍        | 51/361 [46:01<4:11:34, 48.69s/it]
 14%|█▍        | 52/361 [46:51<4:13:08, 49.15s/it]
                                                  
{'loss': 1.5519, 'learning_rate': 0.000156, 'epoch': 0.14}

 14%|█▍        | 52/361 [46:51<4:13:08, 49.15s/it]
 15%|█▍        | 53/361 [47:41<4:13:18, 49.35s/it]
                                                  
{'loss': 1.5892, 'learning_rate': 0.000159, 'epoch': 0.15}

 15%|█▍        | 53/361 [47:41<4:13:18, 49.35s/it]
 15%|█▍        | 54/361 [48:31<4:13:06, 49.47s/it]
                                                  
{'loss': 1.5916, 'learning_rate': 0.000162, 'epoch': 0.15}

 15%|█▍        | 54/361 [48:31<4:13:06, 49.47s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 15%|█▌        | 55/361 [49:17<4:06:59, 48.43s/it]
                                                  
{'loss': 1.5659, 'learning_rate': 0.000165, 'epoch': 0.15}

 15%|█▌        | 55/361 [49:17<4:06:59, 48.43s/it]
 16%|█▌        | 56/361 [50:07<4:08:38, 48.91s/it]
                                                  
{'loss': 1.4988, 'learning_rate': 0.000168, 'epoch': 0.15}

 16%|█▌        | 56/361 [50:07<4:08:38, 48.91s/it]
 16%|█▌        | 57/361 [50:58<4:11:47, 49.70s/it]
                                                  
{'loss': 1.5144, 'learning_rate': 0.00017099999999999998, 'epoch': 0.16}

 16%|█▌        | 57/361 [50:58<4:11:47, 49.70s/it]
 16%|█▌        | 58/361 [51:50<4:14:23, 50.38s/it]
                                                  
{'loss': 1.5053, 'learning_rate': 0.00017399999999999997, 'epoch': 0.16}

 16%|█▌        | 58/361 [51:50<4:14:23, 50.38s/it]
 16%|█▋        | 59/361 [52:40<4:12:34, 50.18s/it]
                                                  
{'loss': 1.4903, 'learning_rate': 0.00017699999999999997, 'epoch': 0.16}

 16%|█▋        | 59/361 [52:40<4:12:34, 50.18s/it]
 17%|█▋        | 60/361 [53:28<4:08:21, 49.51s/it]
                                                  
{'loss': 1.4857, 'learning_rate': 0.00017999999999999998, 'epoch': 0.17}

 17%|█▋        | 60/361 [53:28<4:08:21, 49.51s/it]
 17%|█▋        | 61/361 [54:20<4:10:56, 50.19s/it]
                                                  
{'loss': 1.4456, 'learning_rate': 0.00018299999999999998, 'epoch': 0.17}

 17%|█▋        | 61/361 [54:20<4:10:56, 50.19s/it]
 17%|█▋        | 62/361 [55:10<4:10:08, 50.20s/it]
                                                  
{'loss': 1.5199, 'learning_rate': 0.000186, 'epoch': 0.17}

 17%|█▋        | 62/361 [55:10<4:10:08, 50.20s/it]
 17%|█▋        | 63/361 [55:58<4:06:50, 49.70s/it]
                                                  
{'loss': 1.5311, 'learning_rate': 0.00018899999999999999, 'epoch': 0.17}

 17%|█▋        | 63/361 [55:58<4:06:50, 49.70s/it]
 18%|█▊        | 64/361 [56:49<4:07:34, 50.01s/it]
                                                  
{'loss': 1.5811, 'learning_rate': 0.00019199999999999998, 'epoch': 0.18}

 18%|█▊        | 64/361 [56:49<4:07:34, 50.01s/it]
 18%|█▊        | 65/361 [57:41<4:08:36, 50.39s/it]
                                                  
{'loss': 1.5113, 'learning_rate': 0.000195, 'epoch': 0.18}

 18%|█▊        | 65/361 [57:41<4:08:36, 50.39s/it]
 18%|█▊        | 66/361 [58:29<4:05:29, 49.93s/it]
                                                  
{'loss': 1.4845, 'learning_rate': 0.000198, 'epoch': 0.18}

 18%|█▊        | 66/361 [58:29<4:05:29, 49.93s/it]
 19%|█▊        | 67/361 [59:16<4:00:29, 49.08s/it]
                                                  
{'loss': 1.5994, 'learning_rate': 0.000201, 'epoch': 0.19}

 19%|█▊        | 67/361 [59:16<4:00:29, 49.08s/it]
 19%|█▉        | 68/361 [1:00:02<3:54:25, 48.00s/it]
                                                    
{'loss': 1.4559, 'learning_rate': 0.000204, 'epoch': 0.19}

 19%|█▉        | 68/361 [1:00:02<3:54:25, 48.00s/it]
 19%|█▉        | 69/361 [1:00:51<3:55:13, 48.34s/it]
                                                    
{'loss': 1.4947, 'learning_rate': 0.00020699999999999996, 'epoch': 0.19}

 19%|█▉        | 69/361 [1:00:51<3:55:13, 48.34s/it]
 19%|█▉        | 70/361 [1:01:43<3:59:01, 49.28s/it]
                                                    
{'loss': 1.4919, 'learning_rate': 0.00020999999999999998, 'epoch': 0.19}

 19%|█▉        | 70/361 [1:01:43<3:59:01, 49.28s/it]
 20%|█▉        | 71/361 [1:02:31<3:56:22, 48.90s/it]
                                                    
{'loss': 1.5454, 'learning_rate': 0.00021299999999999997, 'epoch': 0.2}

 20%|█▉        | 71/361 [1:02:31<3:56:22, 48.90s/it]
 20%|█▉        | 72/361 [1:03:17<3:51:35, 48.08s/it]
                                                    
{'loss': 1.5887, 'learning_rate': 0.00021599999999999996, 'epoch': 0.2}

 20%|█▉        | 72/361 [1:03:17<3:51:35, 48.08s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 20%|██        | 73/361 [1:04:07<3:53:58, 48.75s/it]
                                                    
{'loss': 1.5357, 'learning_rate': 0.00021899999999999998, 'epoch': 0.2}

 20%|██        | 73/361 [1:04:07<3:53:58, 48.75s/it]
 20%|██        | 74/361 [1:04:54<3:50:13, 48.13s/it]
                                                    
{'loss': 1.4909, 'learning_rate': 0.00022199999999999998, 'epoch': 0.2}

 20%|██        | 74/361 [1:04:54<3:50:13, 48.13s/it]
 21%|██        | 75/361 [1:05:40<3:46:41, 47.56s/it]
                                                    
{'loss': 1.5885, 'learning_rate': 0.000225, 'epoch': 0.21}

 21%|██        | 75/361 [1:05:40<3:46:41, 47.56s/it]
 21%|██        | 76/361 [1:06:30<3:49:10, 48.25s/it]
                                                    
{'loss': 1.449, 'learning_rate': 0.00022799999999999999, 'epoch': 0.21}

 21%|██        | 76/361 [1:06:30<3:49:10, 48.25s/it]
 21%|██▏       | 77/361 [1:07:20<3:51:43, 48.96s/it]
                                                    
{'loss': 1.5231, 'learning_rate': 0.00023099999999999998, 'epoch': 0.21}

 21%|██▏       | 77/361 [1:07:20<3:51:43, 48.96s/it]
 22%|██▏       | 78/361 [1:08:07<3:47:15, 48.18s/it]
                                                    
{'loss': 1.5615, 'learning_rate': 0.000234, 'epoch': 0.22}

 22%|██▏       | 78/361 [1:08:07<3:47:15, 48.18s/it]
 22%|██▏       | 79/361 [1:08:57<3:49:33, 48.84s/it]
                                                    
{'loss': 1.5125, 'learning_rate': 0.000237, 'epoch': 0.22}

 22%|██▏       | 79/361 [1:08:57<3:49:33, 48.84s/it]
 22%|██▏       | 80/361 [1:09:44<3:46:09, 48.29s/it]
                                                    
{'loss': 1.566, 'learning_rate': 0.00023999999999999998, 'epoch': 0.22}

 22%|██▏       | 80/361 [1:09:44<3:46:09, 48.29s/it]
 22%|██▏       | 81/361 [1:10:35<3:49:20, 49.15s/it]
                                                    
{'loss': 1.4934, 'learning_rate': 0.000243, 'epoch': 0.22}

 22%|██▏       | 81/361 [1:10:35<3:49:20, 49.15s/it]
 23%|██▎       | 82/361 [1:11:23<3:46:28, 48.70s/it]
                                                    
{'loss': 1.5401, 'learning_rate': 0.00024599999999999996, 'epoch': 0.23}

 23%|██▎       | 82/361 [1:11:23<3:46:28, 48.70s/it]
 23%|██▎       | 83/361 [1:12:11<3:44:06, 48.37s/it]
                                                    
{'loss': 1.5093, 'learning_rate': 0.000249, 'epoch': 0.23}

 23%|██▎       | 83/361 [1:12:11<3:44:06, 48.37s/it]
 23%|██▎       | 84/361 [1:13:01<3:46:01, 48.96s/it]
                                                    
{'loss': 1.5854, 'learning_rate': 0.00025199999999999995, 'epoch': 0.23}

 23%|██▎       | 84/361 [1:13:01<3:46:01, 48.96s/it]
 24%|██▎       | 85/361 [1:13:48<3:43:14, 48.53s/it]
                                                    
{'loss': 1.5763, 'learning_rate': 0.00025499999999999996, 'epoch': 0.24}

 24%|██▎       | 85/361 [1:13:48<3:43:14, 48.53s/it]
 24%|██▍       | 86/361 [1:14:36<3:41:46, 48.39s/it]
                                                    
{'loss': 1.4944, 'learning_rate': 0.000258, 'epoch': 0.24}

 24%|██▍       | 86/361 [1:14:36<3:41:46, 48.39s/it]
 24%|██▍       | 87/361 [1:15:28<3:44:45, 49.22s/it]
                                                    
{'loss': 1.4643, 'learning_rate': 0.000261, 'epoch': 0.24}

 24%|██▍       | 87/361 [1:15:28<3:44:45, 49.22s/it]
 24%|██▍       | 88/361 [1:16:15<3:41:48, 48.75s/it]
                                                    
{'loss': 1.5148, 'learning_rate': 0.00026399999999999997, 'epoch': 0.24}

 24%|██▍       | 88/361 [1:16:15<3:41:48, 48.75s/it]
 25%|██▍       | 89/361 [1:17:05<3:42:26, 49.07s/it]
                                                    
{'loss': 1.5205, 'learning_rate': 0.000267, 'epoch': 0.25}

 25%|██▍       | 89/361 [1:17:05<3:42:26, 49.07s/it]
 25%|██▍       | 90/361 [1:17:54<3:41:39, 49.08s/it]
                                                    
{'loss': 1.5123, 'learning_rate': 0.00027, 'epoch': 0.25}

 25%|██▍       | 90/361 [1:17:54<3:41:39, 49.08s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 25%|██▌       | 91/361 [1:18:43<3:40:45, 49.06s/it]
                                                    
{'loss': 1.4845, 'learning_rate': 0.00027299999999999997, 'epoch': 0.25}

 25%|██▌       | 91/361 [1:18:43<3:40:45, 49.06s/it]
 25%|██▌       | 92/361 [1:19:34<3:42:48, 49.70s/it]
                                                    
{'loss': 1.4833, 'learning_rate': 0.000276, 'epoch': 0.25}

 25%|██▌       | 92/361 [1:19:34<3:42:48, 49.70s/it]
 26%|██▌       | 93/361 [1:20:25<3:43:19, 50.00s/it]
                                                    
{'loss': 1.5221, 'learning_rate': 0.000279, 'epoch': 0.26}

 26%|██▌       | 93/361 [1:20:25<3:43:19, 50.00s/it]
 26%|██▌       | 94/361 [1:21:13<3:39:58, 49.43s/it]
                                                    
{'loss': 1.4945, 'learning_rate': 0.00028199999999999997, 'epoch': 0.26}

 26%|██▌       | 94/361 [1:21:13<3:39:58, 49.43s/it]
 26%|██▋       | 95/361 [1:22:03<3:40:06, 49.65s/it]
                                                    
{'loss': 1.5398, 'learning_rate': 0.000285, 'epoch': 0.26}

 26%|██▋       | 95/361 [1:22:03<3:40:06, 49.65s/it]
 27%|██▋       | 96/361 [1:22:52<3:38:13, 49.41s/it]
                                                    
{'loss': 1.4957, 'learning_rate': 0.00028799999999999995, 'epoch': 0.27}

 27%|██▋       | 96/361 [1:22:52<3:38:13, 49.41s/it]
 27%|██▋       | 97/361 [1:23:43<3:39:02, 49.78s/it]
                                                    
{'loss': 1.4619, 'learning_rate': 0.00029099999999999997, 'epoch': 0.27}

 27%|██▋       | 97/361 [1:23:43<3:39:02, 49.78s/it]
 27%|██▋       | 98/361 [1:24:33<3:39:09, 50.00s/it]
                                                    
{'loss': 1.4167, 'learning_rate': 0.000294, 'epoch': 0.27}

 27%|██▋       | 98/361 [1:24:33<3:39:09, 50.00s/it]
 27%|██▋       | 99/361 [1:25:19<3:32:37, 48.69s/it]
                                                    
{'loss': 1.4981, 'learning_rate': 0.00029699999999999996, 'epoch': 0.27}

 27%|██▋       | 99/361 [1:25:19<3:32:37, 48.69s/it]
 28%|██▊       | 100/361 [1:26:09<3:33:28, 49.07s/it]
                                                     
{'loss': 1.4892, 'learning_rate': 0.0003, 'epoch': 0.28}

 28%|██▊       | 100/361 [1:26:09<3:33:28, 49.07s/it]
 28%|██▊       | 101/361 [1:26:58<3:33:09, 49.19s/it]
                                                     
{'loss': 1.5371, 'learning_rate': 0.00029885057471264366, 'epoch': 0.28}

 28%|██▊       | 101/361 [1:26:58<3:33:09, 49.19s/it]
 28%|██▊       | 102/361 [1:27:48<3:32:58, 49.34s/it]
                                                     
{'loss': 1.4711, 'learning_rate': 0.00029770114942528734, 'epoch': 0.28}

 28%|██▊       | 102/361 [1:27:48<3:32:58, 49.34s/it]
 29%|██▊       | 103/361 [1:28:39<3:34:16, 49.83s/it]
                                                     
{'loss': 1.5713, 'learning_rate': 0.00029655172413793103, 'epoch': 0.28}

 29%|██▊       | 103/361 [1:28:39<3:34:16, 49.83s/it]
 29%|██▉       | 104/361 [1:29:26<3:29:43, 48.96s/it]
                                                     
{'loss': 1.571, 'learning_rate': 0.00029540229885057466, 'epoch': 0.29}

 29%|██▉       | 104/361 [1:29:26<3:29:43, 48.96s/it]
 29%|██▉       | 105/361 [1:30:16<3:29:42, 49.15s/it]
                                                     
{'loss': 1.4954, 'learning_rate': 0.00029425287356321835, 'epoch': 0.29}

 29%|██▉       | 105/361 [1:30:16<3:29:42, 49.15s/it]
 29%|██▉       | 106/361 [1:31:05<3:28:51, 49.14s/it]
                                                     
{'loss': 1.4578, 'learning_rate': 0.00029310344827586203, 'epoch': 0.29}

 29%|██▉       | 106/361 [1:31:05<3:28:51, 49.14s/it]
 30%|██▉       | 107/361 [1:31:54<3:27:43, 49.07s/it]
                                                     
{'loss': 1.5052, 'learning_rate': 0.0002919540229885057, 'epoch': 0.3}

 30%|██▉       | 107/361 [1:31:54<3:27:43, 49.07s/it]
 30%|██▉       | 108/361 [1:32:46<3:31:11, 50.09s/it]
                                                     
{'loss': 1.4366, 'learning_rate': 0.0002908045977011494, 'epoch': 0.3}

 30%|██▉       | 108/361 [1:32:46<3:31:11, 50.09s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 30%|███       | 109/361 [1:33:38<3:31:59, 50.47s/it]
                                                     
{'loss': 1.4623, 'learning_rate': 0.0002896551724137931, 'epoch': 0.3}

 30%|███       | 109/361 [1:33:38<3:31:59, 50.47s/it]
 30%|███       | 110/361 [1:34:25<3:27:30, 49.60s/it]
                                                     
{'loss': 1.4894, 'learning_rate': 0.00028850574712643677, 'epoch': 0.3}

 30%|███       | 110/361 [1:34:25<3:27:30, 49.60s/it]
 31%|███       | 111/361 [1:35:12<3:23:48, 48.91s/it]
                                                     
{'loss': 1.4841, 'learning_rate': 0.0002873563218390804, 'epoch': 0.31}

 31%|███       | 111/361 [1:35:12<3:23:48, 48.91s/it]
 31%|███       | 112/361 [1:36:03<3:24:47, 49.35s/it]
                                                     
{'loss': 1.5057, 'learning_rate': 0.0002862068965517241, 'epoch': 0.31}

 31%|███       | 112/361 [1:36:03<3:24:47, 49.35s/it]
 31%|███▏      | 113/361 [1:36:52<3:24:01, 49.36s/it]
                                                     
{'loss': 1.6175, 'learning_rate': 0.00028505747126436777, 'epoch': 0.31}

 31%|███▏      | 113/361 [1:36:52<3:24:01, 49.36s/it]
 32%|███▏      | 114/361 [1:37:41<3:22:50, 49.27s/it]
                                                     
{'loss': 1.48, 'learning_rate': 0.00028390804597701146, 'epoch': 0.32}

 32%|███▏      | 114/361 [1:37:41<3:22:50, 49.27s/it]
 32%|███▏      | 115/361 [1:38:31<3:22:35, 49.41s/it]
                                                     
{'loss': 1.5686, 'learning_rate': 0.00028275862068965514, 'epoch': 0.32}

 32%|███▏      | 115/361 [1:38:31<3:22:35, 49.41s/it]
 32%|███▏      | 116/361 [1:39:19<3:20:34, 49.12s/it]
                                                     
{'loss': 1.5728, 'learning_rate': 0.00028160919540229883, 'epoch': 0.32}

 32%|███▏      | 116/361 [1:39:19<3:20:34, 49.12s/it]
 32%|███▏      | 117/361 [1:40:07<3:18:19, 48.77s/it]
                                                     
{'loss': 1.4764, 'learning_rate': 0.0002804597701149425, 'epoch': 0.32}

 32%|███▏      | 117/361 [1:40:07<3:18:19, 48.77s/it]
 33%|███▎      | 118/361 [1:40:57<3:18:01, 48.90s/it]
                                                     
{'loss': 1.5843, 'learning_rate': 0.0002793103448275862, 'epoch': 0.33}

 33%|███▎      | 118/361 [1:40:57<3:18:01, 48.90s/it]
 33%|███▎      | 119/361 [1:41:46<3:18:31, 49.22s/it]
                                                     
{'loss': 1.465, 'learning_rate': 0.0002781609195402299, 'epoch': 0.33}

 33%|███▎      | 119/361 [1:41:47<3:18:31, 49.22s/it]
 33%|███▎      | 120/361 [1:42:35<3:17:23, 49.14s/it]
                                                     
{'loss': 1.4592, 'learning_rate': 0.00027701149425287357, 'epoch': 0.33}

 33%|███▎      | 120/361 [1:42:35<3:17:23, 49.14s/it]
 34%|███▎      | 121/361 [1:43:24<3:16:16, 49.07s/it]
                                                     
{'loss': 1.4654, 'learning_rate': 0.0002758620689655172, 'epoch': 0.33}

 34%|███▎      | 121/361 [1:43:24<3:16:16, 49.07s/it]
 34%|███▍      | 122/361 [1:44:12<3:14:16, 48.77s/it]
                                                     
{'loss': 1.5443, 'learning_rate': 0.0002747126436781609, 'epoch': 0.34}

 34%|███▍      | 122/361 [1:44:12<3:14:16, 48.77s/it]
 34%|███▍      | 123/361 [1:45:02<3:14:11, 48.96s/it]
                                                     
{'loss': 1.4638, 'learning_rate': 0.00027356321839080457, 'epoch': 0.34}

 34%|███▍      | 123/361 [1:45:02<3:14:11, 48.96s/it]
 34%|███▍      | 124/361 [1:45:53<3:16:04, 49.64s/it]
                                                     
{'loss': 1.5513, 'learning_rate': 0.00027241379310344825, 'epoch': 0.34}

 34%|███▍      | 124/361 [1:45:53<3:16:04, 49.64s/it]
 35%|███▍      | 125/361 [1:46:45<3:18:33, 50.48s/it]
                                                     
{'loss': 1.4068, 'learning_rate': 0.00027126436781609194, 'epoch': 0.35}

 35%|███▍      | 125/361 [1:46:45<3:18:33, 50.48s/it]
 35%|███▍      | 126/361 [1:47:34<3:15:51, 50.01s/it]
                                                     
{'loss': 1.521, 'learning_rate': 0.0002701149425287356, 'epoch': 0.35}

 35%|███▍      | 126/361 [1:47:34<3:15:51, 50.01s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 35%|███▌      | 127/361 [1:48:24<3:14:41, 49.92s/it]
                                                     
{'loss': 1.4559, 'learning_rate': 0.0002689655172413793, 'epoch': 0.35}

 35%|███▌      | 127/361 [1:48:24<3:14:41, 49.92s/it]
 35%|███▌      | 128/361 [1:49:15<3:14:41, 50.14s/it]
                                                     
{'loss': 1.5626, 'learning_rate': 0.00026781609195402294, 'epoch': 0.35}

 35%|███▌      | 128/361 [1:49:15<3:14:41, 50.14s/it]
 36%|███▌      | 129/361 [1:50:04<3:12:47, 49.86s/it]
                                                     
{'loss': 1.5089, 'learning_rate': 0.0002666666666666666, 'epoch': 0.36}

 36%|███▌      | 129/361 [1:50:04<3:12:47, 49.86s/it]
 36%|███▌      | 130/361 [1:50:55<3:13:11, 50.18s/it]
                                                     
{'loss': 1.4241, 'learning_rate': 0.0002655172413793103, 'epoch': 0.36}

 36%|███▌      | 130/361 [1:50:55<3:13:11, 50.18s/it]
 36%|███▋      | 131/361 [1:51:44<3:10:41, 49.74s/it]
                                                     
{'loss': 1.5163, 'learning_rate': 0.000264367816091954, 'epoch': 0.36}

 36%|███▋      | 131/361 [1:51:44<3:10:41, 49.74s/it]
 37%|███▋      | 132/361 [1:52:32<3:07:52, 49.22s/it]
                                                     
{'loss': 1.481, 'learning_rate': 0.0002632183908045977, 'epoch': 0.37}

 37%|███▋      | 132/361 [1:52:32<3:07:52, 49.22s/it]
 37%|███▋      | 133/361 [1:53:21<3:07:17, 49.29s/it]
                                                     
{'loss': 1.4552, 'learning_rate': 0.00026206896551724137, 'epoch': 0.37}

 37%|███▋      | 133/361 [1:53:21<3:07:17, 49.29s/it]
 37%|███▋      | 134/361 [1:54:08<3:03:16, 48.44s/it]
                                                     
{'loss': 1.4643, 'learning_rate': 0.00026091954022988505, 'epoch': 0.37}

 37%|███▋      | 134/361 [1:54:08<3:03:16, 48.44s/it]
 37%|███▋      | 135/361 [1:55:00<3:06:38, 49.55s/it]
                                                     
{'loss': 1.4456, 'learning_rate': 0.0002597701149425287, 'epoch': 0.37}

 37%|███▋      | 135/361 [1:55:00<3:06:38, 49.55s/it]
 38%|███▊      | 136/361 [1:55:48<3:03:56, 49.05s/it]
                                                     
{'loss': 1.5658, 'learning_rate': 0.00025862068965517237, 'epoch': 0.38}

 38%|███▊      | 136/361 [1:55:48<3:03:56, 49.05s/it]
 38%|███▊      | 137/361 [1:56:36<3:02:38, 48.92s/it]
                                                     
{'loss': 1.5064, 'learning_rate': 0.00025747126436781605, 'epoch': 0.38}

 38%|███▊      | 137/361 [1:56:36<3:02:38, 48.92s/it]
 38%|███▊      | 138/361 [1:57:26<3:02:36, 49.13s/it]
                                                     
{'loss': 1.5429, 'learning_rate': 0.00025632183908045974, 'epoch': 0.38}

 38%|███▊      | 138/361 [1:57:26<3:02:36, 49.13s/it]
 39%|███▊      | 139/361 [1:58:14<3:00:44, 48.85s/it]
                                                     
{'loss': 1.5054, 'learning_rate': 0.0002551724137931034, 'epoch': 0.38}

 39%|███▊      | 139/361 [1:58:14<3:00:44, 48.85s/it]
 39%|███▉      | 140/361 [1:59:02<2:59:23, 48.70s/it]
                                                     
{'loss': 1.4889, 'learning_rate': 0.0002540229885057471, 'epoch': 0.39}

 39%|███▉      | 140/361 [1:59:02<2:59:23, 48.70s/it]
 39%|███▉      | 141/361 [1:59:49<2:56:17, 48.08s/it]
                                                     
{'loss': 1.4501, 'learning_rate': 0.0002528735632183908, 'epoch': 0.39}

 39%|███▉      | 141/361 [1:59:49<2:56:17, 48.08s/it]
 39%|███▉      | 142/361 [2:00:39<2:57:51, 48.73s/it]
                                                     
{'loss': 1.4571, 'learning_rate': 0.0002517241379310345, 'epoch': 0.39}

 39%|███▉      | 142/361 [2:00:39<2:57:51, 48.73s/it]
 40%|███▉      | 143/361 [2:01:26<2:55:15, 48.24s/it]
                                                     
{'loss': 1.4824, 'learning_rate': 0.00025057471264367816, 'epoch': 0.4}

 40%|███▉      | 143/361 [2:01:26<2:55:15, 48.24s/it]
 40%|███▉      | 144/361 [2:02:18<2:58:20, 49.31s/it]
                                                     
{'loss': 1.4365, 'learning_rate': 0.00024942528735632185, 'epoch': 0.4}

 40%|███▉      | 144/361 [2:02:18<2:58:20, 49.31s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 40%|████      | 145/361 [2:03:10<3:00:04, 50.02s/it]
                                                     
{'loss': 1.4284, 'learning_rate': 0.0002482758620689655, 'epoch': 0.4}

 40%|████      | 145/361 [2:03:10<3:00:04, 50.02s/it]
 40%|████      | 146/361 [2:04:02<3:01:15, 50.58s/it]
                                                     
{'loss': 1.4696, 'learning_rate': 0.00024712643678160916, 'epoch': 0.4}

 40%|████      | 146/361 [2:04:02<3:01:15, 50.58s/it]
 41%|████      | 147/361 [2:04:54<3:02:19, 51.12s/it]
                                                     
{'loss': 1.5034, 'learning_rate': 0.00024597701149425285, 'epoch': 0.41}

 41%|████      | 147/361 [2:04:54<3:02:19, 51.12s/it]
 41%|████      | 148/361 [2:05:45<3:00:45, 50.92s/it]
                                                     
{'loss': 1.5225, 'learning_rate': 0.00024482758620689653, 'epoch': 0.41}

 41%|████      | 148/361 [2:05:45<3:00:45, 50.92s/it]
 41%|████▏     | 149/361 [2:06:37<3:01:39, 51.41s/it]
                                                     
{'loss': 1.4808, 'learning_rate': 0.0002436781609195402, 'epoch': 0.41}

 41%|████▏     | 149/361 [2:06:37<3:01:39, 51.41s/it]
 42%|████▏     | 150/361 [2:07:27<2:59:31, 51.05s/it]
                                                     
{'loss': 1.4693, 'learning_rate': 0.0002425287356321839, 'epoch': 0.41}

 42%|████▏     | 150/361 [2:07:27<2:59:31, 51.05s/it]
 42%|████▏     | 151/361 [2:08:18<2:58:39, 51.04s/it]
                                                     
{'loss': 1.548, 'learning_rate': 0.00024137931034482756, 'epoch': 0.42}

 42%|████▏     | 151/361 [2:08:18<2:58:39, 51.04s/it]
 42%|████▏     | 152/361 [2:09:08<2:56:47, 50.75s/it]
                                                     
{'loss': 1.4709, 'learning_rate': 0.00024022988505747125, 'epoch': 0.42}

 42%|████▏     | 152/361 [2:09:08<2:56:47, 50.75s/it]
 42%|████▏     | 153/361 [2:09:55<2:51:55, 49.59s/it]
                                                     
{'loss': 1.5187, 'learning_rate': 0.0002390804597701149, 'epoch': 0.42}

 42%|████▏     | 153/361 [2:09:55<2:51:55, 49.59s/it]
 43%|████▎     | 154/361 [2:10:47<2:52:57, 50.13s/it]
                                                     
{'loss': 1.4951, 'learning_rate': 0.00023793103448275862, 'epoch': 0.43}

 43%|████▎     | 154/361 [2:10:47<2:52:57, 50.13s/it]
 43%|████▎     | 155/361 [2:11:36<2:51:02, 49.82s/it]
                                                     
{'loss': 1.4738, 'learning_rate': 0.00023678160919540228, 'epoch': 0.43}

 43%|████▎     | 155/361 [2:11:36<2:51:02, 49.82s/it]
 43%|████▎     | 156/361 [2:12:25<2:49:23, 49.58s/it]
                                                     
{'loss': 1.3997, 'learning_rate': 0.00023563218390804593, 'epoch': 0.43}

 43%|████▎     | 156/361 [2:12:25<2:49:23, 49.58s/it]
 43%|████▎     | 157/361 [2:13:17<2:51:36, 50.47s/it]
                                                     
{'loss': 1.3933, 'learning_rate': 0.00023448275862068965, 'epoch': 0.43}

 43%|████▎     | 157/361 [2:13:17<2:51:36, 50.47s/it]
 44%|████▍     | 158/361 [2:14:04<2:47:18, 49.45s/it]
                                                     
{'loss': 1.4621, 'learning_rate': 0.0002333333333333333, 'epoch': 0.44}

 44%|████▍     | 158/361 [2:14:04<2:47:18, 49.45s/it]
 44%|████▍     | 159/361 [2:14:55<2:47:45, 49.83s/it]
                                                     
{'loss': 1.5174, 'learning_rate': 0.000232183908045977, 'epoch': 0.44}

 44%|████▍     | 159/361 [2:14:55<2:47:45, 49.83s/it]
 44%|████▍     | 160/361 [2:15:42<2:43:35, 48.83s/it]
                                                     
{'loss': 1.4443, 'learning_rate': 0.00023103448275862065, 'epoch': 0.44}

 44%|████▍     | 160/361 [2:15:42<2:43:35, 48.83s/it]
 45%|████▍     | 161/361 [2:16:32<2:43:52, 49.16s/it]
                                                     
{'loss': 1.4707, 'learning_rate': 0.00022988505747126436, 'epoch': 0.45}

 45%|████▍     | 161/361 [2:16:32<2:43:52, 49.16s/it]
 45%|████▍     | 162/361 [2:17:22<2:43:53, 49.41s/it]
                                                     
{'loss': 1.4664, 'learning_rate': 0.00022873563218390802, 'epoch': 0.45}

 45%|████▍     | 162/361 [2:17:22<2:43:53, 49.41s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 45%|████▌     | 163/361 [2:18:10<2:41:42, 49.00s/it]
                                                     
{'loss': 1.4772, 'learning_rate': 0.0002275862068965517, 'epoch': 0.45}

 45%|████▌     | 163/361 [2:18:10<2:41:42, 49.00s/it]
 45%|████▌     | 164/361 [2:18:58<2:40:17, 48.82s/it]
                                                     
{'loss': 1.4981, 'learning_rate': 0.0002264367816091954, 'epoch': 0.45}

 45%|████▌     | 164/361 [2:18:58<2:40:17, 48.82s/it]
 46%|████▌     | 165/361 [2:19:50<2:42:46, 49.83s/it]
                                                     
{'loss': 1.4877, 'learning_rate': 0.00022528735632183907, 'epoch': 0.46}

 46%|████▌     | 165/361 [2:19:50<2:42:46, 49.83s/it]
 46%|████▌     | 166/361 [2:20:41<2:43:12, 50.22s/it]
                                                     
{'loss': 1.452, 'learning_rate': 0.00022413793103448273, 'epoch': 0.46}

 46%|████▌     | 166/361 [2:20:41<2:43:12, 50.22s/it]
 46%|████▋     | 167/361 [2:21:28<2:38:45, 49.10s/it]
                                                     
{'loss': 1.5105, 'learning_rate': 0.00022298850574712644, 'epoch': 0.46}

 46%|████▋     | 167/361 [2:21:28<2:38:45, 49.10s/it]
 47%|████▋     | 168/361 [2:22:16<2:36:44, 48.73s/it]
                                                     
{'loss': 1.3836, 'learning_rate': 0.0002218390804597701, 'epoch': 0.46}

 47%|████▋     | 168/361 [2:22:16<2:36:44, 48.73s/it]
 47%|████▋     | 169/361 [2:23:10<2:41:39, 50.52s/it]
                                                     
{'loss': 1.4118, 'learning_rate': 0.00022068965517241376, 'epoch': 0.47}

 47%|████▋     | 169/361 [2:23:10<2:41:39, 50.52s/it]
 47%|████▋     | 170/361 [2:23:59<2:38:46, 49.88s/it]
                                                     
{'loss': 1.5386, 'learning_rate': 0.00021954022988505744, 'epoch': 0.47}

 47%|████▋     | 170/361 [2:23:59<2:38:46, 49.88s/it]
 47%|████▋     | 171/361 [2:24:46<2:35:49, 49.21s/it]
                                                     
{'loss': 1.4262, 'learning_rate': 0.00021839080459770113, 'epoch': 0.47}

 47%|████▋     | 171/361 [2:24:46<2:35:49, 49.21s/it]
 48%|████▊     | 172/361 [2:25:36<2:35:47, 49.46s/it]
                                                     
{'loss': 1.4009, 'learning_rate': 0.00021724137931034481, 'epoch': 0.48}

 48%|████▊     | 172/361 [2:25:36<2:35:47, 49.46s/it]
 48%|████▊     | 173/361 [2:26:25<2:34:36, 49.34s/it]
                                                     
{'loss': 1.5275, 'learning_rate': 0.00021609195402298847, 'epoch': 0.48}

 48%|████▊     | 173/361 [2:26:25<2:34:36, 49.34s/it]
 48%|████▊     | 174/361 [2:27:15<2:34:01, 49.42s/it]
                                                     
{'loss': 1.5311, 'learning_rate': 0.00021494252873563218, 'epoch': 0.48}

 48%|████▊     | 174/361 [2:27:15<2:34:01, 49.42s/it]
 48%|████▊     | 175/361 [2:28:06<2:34:32, 49.85s/it]
                                                     
{'loss': 1.5075, 'learning_rate': 0.00021379310344827584, 'epoch': 0.48}

 48%|████▊     | 175/361 [2:28:06<2:34:32, 49.85s/it]
 49%|████▉     | 176/361 [2:28:56<2:34:13, 50.02s/it]
                                                     
{'loss': 1.4087, 'learning_rate': 0.00021264367816091953, 'epoch': 0.49}

 49%|████▉     | 176/361 [2:28:56<2:34:13, 50.02s/it]
 49%|████▉     | 177/361 [2:29:46<2:33:19, 50.00s/it]
                                                     
{'loss': 1.5531, 'learning_rate': 0.00021149425287356319, 'epoch': 0.49}

 49%|████▉     | 177/361 [2:29:46<2:33:19, 50.00s/it]
 49%|████▉     | 178/361 [2:30:33<2:29:07, 48.89s/it]
                                                     
{'loss': 1.5489, 'learning_rate': 0.0002103448275862069, 'epoch': 0.49}

 49%|████▉     | 178/361 [2:30:33<2:29:07, 48.89s/it]
 50%|████▉     | 179/361 [2:31:23<2:29:55, 49.43s/it]
                                                     
{'loss': 1.4843, 'learning_rate': 0.00020919540229885056, 'epoch': 0.49}

 50%|████▉     | 179/361 [2:31:23<2:29:55, 49.43s/it]
 50%|████▉     | 180/361 [2:32:12<2:28:46, 49.32s/it]
                                                     
{'loss': 1.5217, 'learning_rate': 0.00020804597701149421, 'epoch': 0.5}

 50%|████▉     | 180/361 [2:32:12<2:28:46, 49.32s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 50%|█████     | 181/361 [2:33:01<2:27:29, 49.16s/it]
                                                     
{'loss': 1.4821, 'learning_rate': 0.00020689655172413793, 'epoch': 0.5}

 50%|█████     | 181/361 [2:33:01<2:27:29, 49.16s/it]
 50%|█████     | 182/361 [2:33:52<2:27:54, 49.58s/it]
                                                     
{'loss': 1.4693, 'learning_rate': 0.00020574712643678158, 'epoch': 0.5}

 50%|█████     | 182/361 [2:33:52<2:27:54, 49.58s/it]
 51%|█████     | 183/361 [2:34:45<2:30:22, 50.69s/it]
                                                     
{'loss': 1.4187, 'learning_rate': 0.00020459770114942527, 'epoch': 0.51}

 51%|█████     | 183/361 [2:34:45<2:30:22, 50.69s/it]
 51%|█████     | 184/361 [2:35:35<2:28:32, 50.35s/it]
                                                     
{'loss': 1.5126, 'learning_rate': 0.00020344827586206893, 'epoch': 0.51}

 51%|█████     | 184/361 [2:35:35<2:28:32, 50.35s/it]
 51%|█████     | 185/361 [2:36:22<2:25:34, 49.63s/it]
                                                     
{'loss': 1.51, 'learning_rate': 0.00020229885057471264, 'epoch': 0.51}

 51%|█████     | 185/361 [2:36:22<2:25:34, 49.63s/it]
 52%|█████▏    | 186/361 [2:37:13<2:25:06, 49.75s/it]
                                                     
{'loss': 1.4563, 'learning_rate': 0.0002011494252873563, 'epoch': 0.51}

 52%|█████▏    | 186/361 [2:37:13<2:25:06, 49.75s/it]
 52%|█████▏    | 187/361 [2:38:02<2:24:26, 49.81s/it]
                                                     
{'loss': 1.5515, 'learning_rate': 0.00019999999999999998, 'epoch': 0.52}

 52%|█████▏    | 187/361 [2:38:02<2:24:26, 49.81s/it]
 52%|█████▏    | 188/361 [2:38:53<2:24:27, 50.10s/it]
                                                     
{'loss': 1.4553, 'learning_rate': 0.00019885057471264367, 'epoch': 0.52}

 52%|█████▏    | 188/361 [2:38:53<2:24:27, 50.10s/it]
 52%|█████▏    | 189/361 [2:39:42<2:22:36, 49.75s/it]
                                                     
{'loss': 1.4807, 'learning_rate': 0.00019770114942528735, 'epoch': 0.52}

 52%|█████▏    | 189/361 [2:39:42<2:22:36, 49.75s/it]
 53%|█████▎    | 190/361 [2:40:31<2:21:06, 49.51s/it]
                                                     
{'loss': 1.4385, 'learning_rate': 0.000196551724137931, 'epoch': 0.53}

 53%|█████▎    | 190/361 [2:40:31<2:21:06, 49.51s/it]
 53%|█████▎    | 191/361 [2:41:20<2:20:04, 49.44s/it]
                                                     
{'loss': 1.4843, 'learning_rate': 0.00019540229885057467, 'epoch': 0.53}

 53%|█████▎    | 191/361 [2:41:20<2:20:04, 49.44s/it]
 53%|█████▎    | 192/361 [2:42:10<2:19:22, 49.48s/it]
                                                     
{'loss': 1.4873, 'learning_rate': 0.00019425287356321838, 'epoch': 0.53}

 53%|█████▎    | 192/361 [2:42:10<2:19:22, 49.48s/it]
 53%|█████▎    | 193/361 [2:42:58<2:17:37, 49.15s/it]
                                                     
{'loss': 1.4772, 'learning_rate': 0.00019310344827586204, 'epoch': 0.53}

 53%|█████▎    | 193/361 [2:42:58<2:17:37, 49.15s/it]
 54%|█████▎    | 194/361 [2:43:49<2:17:39, 49.46s/it]
                                                     
{'loss': 1.4775, 'learning_rate': 0.00019195402298850572, 'epoch': 0.54}

 54%|█████▎    | 194/361 [2:43:49<2:17:39, 49.46s/it]
 54%|█████▍    | 195/361 [2:44:37<2:15:47, 49.08s/it]
                                                     
{'loss': 1.508, 'learning_rate': 0.0001908045977011494, 'epoch': 0.54}

 54%|█████▍    | 195/361 [2:44:37<2:15:47, 49.08s/it]
 54%|█████▍    | 196/361 [2:45:26<2:15:29, 49.27s/it]
                                                     
{'loss': 1.4055, 'learning_rate': 0.0001896551724137931, 'epoch': 0.54}

 54%|█████▍    | 196/361 [2:45:26<2:15:29, 49.27s/it]
 55%|█████▍    | 197/361 [2:46:15<2:14:21, 49.16s/it]
                                                     
{'loss': 1.4829, 'learning_rate': 0.00018850574712643675, 'epoch': 0.54}

 55%|█████▍    | 197/361 [2:46:15<2:14:21, 49.16s/it]
 55%|█████▍    | 198/361 [2:47:04<2:13:26, 49.12s/it]
                                                     
{'loss': 1.4681, 'learning_rate': 0.00018735632183908046, 'epoch': 0.55}

 55%|█████▍    | 198/361 [2:47:04<2:13:26, 49.12s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 55%|█████▌    | 199/361 [2:47:53<2:12:36, 49.11s/it]
                                                     
{'loss': 1.4794, 'learning_rate': 0.00018620689655172412, 'epoch': 0.55}

 55%|█████▌    | 199/361 [2:47:53<2:12:36, 49.11s/it]
 55%|█████▌    | 200/361 [2:48:46<2:14:15, 50.03s/it]
                                                     
{'loss': 1.4204, 'learning_rate': 0.0001850574712643678, 'epoch': 0.55}

 55%|█████▌    | 200/361 [2:48:46<2:14:15, 50.03s/it]
 56%|█████▌    | 201/361 [2:49:32<2:10:44, 49.03s/it]
                                                     
{'loss': 1.4945, 'learning_rate': 0.00018390804597701147, 'epoch': 0.56}

 56%|█████▌    | 201/361 [2:49:32<2:10:44, 49.03s/it]
 56%|█████▌    | 202/361 [2:50:20<2:09:13, 48.76s/it]
                                                     
{'loss': 1.4344, 'learning_rate': 0.00018275862068965518, 'epoch': 0.56}

 56%|█████▌    | 202/361 [2:50:20<2:09:13, 48.76s/it]
 56%|█████▌    | 203/361 [2:51:10<2:08:45, 48.89s/it]
                                                     
{'loss': 1.4005, 'learning_rate': 0.00018160919540229884, 'epoch': 0.56}

 56%|█████▌    | 203/361 [2:51:10<2:08:45, 48.89s/it]
 57%|█████▋    | 204/361 [2:51:59<2:08:26, 49.09s/it]
                                                     
{'loss': 1.494, 'learning_rate': 0.0001804597701149425, 'epoch': 0.56}

 57%|█████▋    | 204/361 [2:51:59<2:08:26, 49.09s/it]
 57%|█████▋    | 205/361 [2:52:48<2:07:20, 48.98s/it]
                                                     
{'loss': 1.5266, 'learning_rate': 0.0001793103448275862, 'epoch': 0.57}

 57%|█████▋    | 205/361 [2:52:48<2:07:20, 48.98s/it]
 57%|█████▋    | 206/361 [2:53:36<2:05:45, 48.68s/it]
                                                     
{'loss': 1.5335, 'learning_rate': 0.00017816091954022986, 'epoch': 0.57}

 57%|█████▋    | 206/361 [2:53:36<2:05:45, 48.68s/it]
 57%|█████▋    | 207/361 [2:54:26<2:06:11, 49.16s/it]
                                                     
{'loss': 1.4804, 'learning_rate': 0.00017701149425287355, 'epoch': 0.57}

 57%|█████▋    | 207/361 [2:54:26<2:06:11, 49.16s/it]
 58%|█████▊    | 208/361 [2:55:17<2:06:50, 49.75s/it]
                                                     
{'loss': 1.526, 'learning_rate': 0.0001758620689655172, 'epoch': 0.58}

 58%|█████▊    | 208/361 [2:55:17<2:06:50, 49.75s/it]
 58%|█████▊    | 209/361 [2:56:06<2:05:31, 49.55s/it]
                                                     
{'loss': 1.4942, 'learning_rate': 0.00017471264367816092, 'epoch': 0.58}

 58%|█████▊    | 209/361 [2:56:06<2:05:31, 49.55s/it]
 58%|█████▊    | 210/361 [2:56:54<2:03:19, 49.01s/it]
                                                     
{'loss': 1.5255, 'learning_rate': 0.00017356321839080458, 'epoch': 0.58}

 58%|█████▊    | 210/361 [2:56:54<2:03:19, 49.01s/it]
 58%|█████▊    | 211/361 [2:57:44<2:03:19, 49.33s/it]
                                                     
{'loss': 1.3654, 'learning_rate': 0.00017241379310344826, 'epoch': 0.58}

 58%|█████▊    | 211/361 [2:57:44<2:03:19, 49.33s/it]
 59%|█████▊    | 212/361 [2:58:33<2:02:07, 49.18s/it]
                                                     
{'loss': 1.5555, 'learning_rate': 0.00017126436781609195, 'epoch': 0.59}

 59%|█████▊    | 212/361 [2:58:33<2:02:07, 49.18s/it]
 59%|█████▉    | 213/361 [2:59:24<2:02:22, 49.61s/it]
                                                     
{'loss': 1.5333, 'learning_rate': 0.00017011494252873563, 'epoch': 0.59}

 59%|█████▉    | 213/361 [2:59:24<2:02:22, 49.61s/it]
 59%|█████▉    | 214/361 [3:00:13<2:01:37, 49.64s/it]
                                                     
{'loss': 1.4874, 'learning_rate': 0.0001689655172413793, 'epoch': 0.59}

 59%|█████▉    | 214/361 [3:00:13<2:01:37, 49.64s/it]
 60%|█████▉    | 215/361 [3:01:04<2:01:31, 49.94s/it]
                                                     
{'loss': 1.4864, 'learning_rate': 0.00016781609195402295, 'epoch': 0.59}

 60%|█████▉    | 215/361 [3:01:04<2:01:31, 49.94s/it]
 60%|█████▉    | 216/361 [3:01:54<2:00:59, 50.07s/it]
                                                     
{'loss': 1.4444, 'learning_rate': 0.00016666666666666666, 'epoch': 0.6}

 60%|█████▉    | 216/361 [3:01:54<2:00:59, 50.07s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 60%|██████    | 217/361 [3:02:42<1:58:24, 49.33s/it]
                                                     
{'loss': 1.4194, 'learning_rate': 0.00016551724137931032, 'epoch': 0.6}

 60%|██████    | 217/361 [3:02:42<1:58:24, 49.33s/it]
 60%|██████    | 218/361 [3:03:34<1:59:07, 49.98s/it]
                                                     
{'loss': 1.4936, 'learning_rate': 0.000164367816091954, 'epoch': 0.6}

 60%|██████    | 218/361 [3:03:34<1:59:07, 49.98s/it]
 61%|██████    | 219/361 [3:04:24<1:58:47, 50.19s/it]
                                                     
{'loss': 1.4436, 'learning_rate': 0.0001632183908045977, 'epoch': 0.61}

 61%|██████    | 219/361 [3:04:24<1:58:47, 50.19s/it]
 61%|██████    | 220/361 [3:05:16<1:59:19, 50.78s/it]
                                                     
{'loss': 1.4461, 'learning_rate': 0.00016206896551724137, 'epoch': 0.61}

 61%|██████    | 220/361 [3:05:16<1:59:19, 50.78s/it]
 61%|██████    | 221/361 [3:06:04<1:56:31, 49.94s/it]
                                                     
{'loss': 1.5182, 'learning_rate': 0.00016091954022988503, 'epoch': 0.61}

 61%|██████    | 221/361 [3:06:04<1:56:31, 49.94s/it]
 61%|██████▏   | 222/361 [3:06:53<1:54:40, 49.50s/it]
                                                     
{'loss': 1.4573, 'learning_rate': 0.00015977011494252874, 'epoch': 0.61}

 61%|██████▏   | 222/361 [3:06:53<1:54:40, 49.50s/it]
 62%|██████▏   | 223/361 [3:07:41<1:53:11, 49.21s/it]
                                                     
{'loss': 1.4392, 'learning_rate': 0.0001586206896551724, 'epoch': 0.62}

 62%|██████▏   | 223/361 [3:07:41<1:53:11, 49.21s/it]
 62%|██████▏   | 224/361 [3:08:31<1:52:48, 49.40s/it]
                                                     
{'loss': 1.445, 'learning_rate': 0.0001574712643678161, 'epoch': 0.62}

 62%|██████▏   | 224/361 [3:08:31<1:52:48, 49.40s/it]
 62%|██████▏   | 225/361 [3:09:22<1:52:51, 49.79s/it]
                                                     
{'loss': 1.3872, 'learning_rate': 0.00015632183908045975, 'epoch': 0.62}

 62%|██████▏   | 225/361 [3:09:22<1:52:51, 49.79s/it]
 63%|██████▎   | 226/361 [3:10:14<1:53:16, 50.35s/it]
                                                     
{'loss': 1.4088, 'learning_rate': 0.00015517241379310346, 'epoch': 0.62}

 63%|██████▎   | 226/361 [3:10:14<1:53:16, 50.35s/it]
 63%|██████▎   | 227/361 [3:11:05<1:53:15, 50.71s/it]
                                                     
{'loss': 1.4938, 'learning_rate': 0.00015402298850574712, 'epoch': 0.63}

 63%|██████▎   | 227/361 [3:11:05<1:53:15, 50.71s/it]
 63%|██████▎   | 228/361 [3:11:55<1:51:33, 50.33s/it]
                                                     
{'loss': 1.5016, 'learning_rate': 0.00015287356321839077, 'epoch': 0.63}

 63%|██████▎   | 228/361 [3:11:55<1:51:33, 50.33s/it]
 63%|██████▎   | 229/361 [3:12:45<1:51:02, 50.47s/it]
                                                     
{'loss': 1.4741, 'learning_rate': 0.00015172413793103449, 'epoch': 0.63}

 63%|██████▎   | 229/361 [3:12:45<1:51:02, 50.47s/it]
 64%|██████▎   | 230/361 [3:13:36<1:50:21, 50.54s/it]
                                                     
{'loss': 1.4637, 'learning_rate': 0.00015057471264367814, 'epoch': 0.64}

 64%|██████▎   | 230/361 [3:13:36<1:50:21, 50.54s/it]
 64%|██████▍   | 231/361 [3:14:26<1:48:50, 50.24s/it]
                                                     
{'loss': 1.4614, 'learning_rate': 0.00014942528735632183, 'epoch': 0.64}

 64%|██████▍   | 231/361 [3:14:26<1:48:50, 50.24s/it]
 64%|██████▍   | 232/361 [3:15:16<1:47:50, 50.16s/it]
                                                     
{'loss': 1.4521, 'learning_rate': 0.00014827586206896551, 'epoch': 0.64}

 64%|██████▍   | 232/361 [3:15:16<1:47:50, 50.16s/it]
 65%|██████▍   | 233/361 [3:16:07<1:47:53, 50.58s/it]
                                                     
{'loss': 1.4901, 'learning_rate': 0.00014712643678160917, 'epoch': 0.64}

 65%|██████▍   | 233/361 [3:16:07<1:47:53, 50.58s/it]
 65%|██████▍   | 234/361 [3:16:57<1:46:50, 50.48s/it]
                                                     
{'loss': 1.5006, 'learning_rate': 0.00014597701149425286, 'epoch': 0.65}

 65%|██████▍   | 234/361 [3:16:57<1:46:50, 50.48s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 65%|██████▌   | 235/361 [3:17:48<1:46:13, 50.59s/it]
                                                     
{'loss': 1.4431, 'learning_rate': 0.00014482758620689654, 'epoch': 0.65}

 65%|██████▌   | 235/361 [3:17:48<1:46:13, 50.59s/it]
 65%|██████▌   | 236/361 [3:18:37<1:44:14, 50.04s/it]
                                                     
{'loss': 1.4558, 'learning_rate': 0.0001436781609195402, 'epoch': 0.65}

 65%|██████▌   | 236/361 [3:18:37<1:44:14, 50.04s/it]
 66%|██████▌   | 237/361 [3:19:28<1:44:18, 50.47s/it]
                                                     
{'loss': 1.453, 'learning_rate': 0.00014252873563218389, 'epoch': 0.66}

 66%|██████▌   | 237/361 [3:19:28<1:44:18, 50.47s/it]
 66%|██████▌   | 238/361 [3:20:18<1:42:47, 50.15s/it]
                                                     
{'loss': 1.3849, 'learning_rate': 0.00014137931034482757, 'epoch': 0.66}

 66%|██████▌   | 238/361 [3:20:18<1:42:47, 50.15s/it]
 66%|██████▌   | 239/361 [3:21:06<1:41:00, 49.68s/it]
                                                     
{'loss': 1.4283, 'learning_rate': 0.00014022988505747126, 'epoch': 0.66}

 66%|██████▌   | 239/361 [3:21:06<1:41:00, 49.68s/it]
 66%|██████▋   | 240/361 [3:21:56<1:40:10, 49.68s/it]
                                                     
{'loss': 1.4855, 'learning_rate': 0.00013908045977011494, 'epoch': 0.66}

 66%|██████▋   | 240/361 [3:21:56<1:40:10, 49.68s/it]
 67%|██████▋   | 241/361 [3:22:46<1:39:34, 49.78s/it]
                                                     
{'loss': 1.4136, 'learning_rate': 0.0001379310344827586, 'epoch': 0.67}

 67%|██████▋   | 241/361 [3:22:46<1:39:34, 49.78s/it]
 67%|██████▋   | 242/361 [3:23:37<1:39:39, 50.25s/it]
                                                     
{'loss': 1.4289, 'learning_rate': 0.00013678160919540228, 'epoch': 0.67}

 67%|██████▋   | 242/361 [3:23:37<1:39:39, 50.25s/it]
 67%|██████▋   | 243/361 [3:24:27<1:38:15, 49.96s/it]
                                                     
{'loss': 1.4104, 'learning_rate': 0.00013563218390804597, 'epoch': 0.67}

 67%|██████▋   | 243/361 [3:24:27<1:38:15, 49.96s/it]
 68%|██████▊   | 244/361 [3:25:14<1:35:40, 49.06s/it]
                                                     
{'loss': 1.4662, 'learning_rate': 0.00013448275862068965, 'epoch': 0.67}

 68%|██████▊   | 244/361 [3:25:14<1:35:40, 49.06s/it]
 68%|██████▊   | 245/361 [3:26:07<1:37:24, 50.38s/it]
                                                     
{'loss': 1.4217, 'learning_rate': 0.0001333333333333333, 'epoch': 0.68}

 68%|██████▊   | 245/361 [3:26:07<1:37:24, 50.38s/it]
 68%|██████▊   | 246/361 [3:26:58<1:36:58, 50.59s/it]
                                                     
{'loss': 1.4359, 'learning_rate': 0.000132183908045977, 'epoch': 0.68}

 68%|██████▊   | 246/361 [3:26:58<1:36:58, 50.59s/it]
 68%|██████▊   | 247/361 [3:27:51<1:37:20, 51.24s/it]
                                                     
{'loss': 1.4623, 'learning_rate': 0.00013103448275862068, 'epoch': 0.68}

 68%|██████▊   | 247/361 [3:27:51<1:37:20, 51.24s/it]
 69%|██████▊   | 248/361 [3:28:42<1:36:25, 51.20s/it]
                                                     
{'loss': 1.382, 'learning_rate': 0.00012988505747126434, 'epoch': 0.69}

 69%|██████▊   | 248/361 [3:28:42<1:36:25, 51.20s/it]
 69%|██████▉   | 249/361 [3:29:33<1:35:19, 51.06s/it]
                                                     
{'loss': 1.3784, 'learning_rate': 0.00012873563218390803, 'epoch': 0.69}

 69%|██████▉   | 249/361 [3:29:33<1:35:19, 51.06s/it]
 69%|██████▉   | 250/361 [3:30:24<1:34:35, 51.13s/it]
                                                     
{'loss': 1.3897, 'learning_rate': 0.0001275862068965517, 'epoch': 0.69}

 69%|██████▉   | 250/361 [3:30:24<1:34:35, 51.13s/it]
 70%|██████▉   | 251/361 [3:31:14<1:33:07, 50.80s/it]
                                                     
{'loss': 1.4414, 'learning_rate': 0.0001264367816091954, 'epoch': 0.69}

 70%|██████▉   | 251/361 [3:31:14<1:33:07, 50.80s/it]
 70%|██████▉   | 252/361 [3:32:05<1:32:30, 50.93s/it]
                                                     
{'loss': 1.427, 'learning_rate': 0.00012528735632183908, 'epoch': 0.7}

 70%|██████▉   | 252/361 [3:32:05<1:32:30, 50.93s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 70%|███████   | 253/361 [3:32:55<1:30:48, 50.45s/it]
                                                     
{'loss': 1.4165, 'learning_rate': 0.00012413793103448274, 'epoch': 0.7}

 70%|███████   | 253/361 [3:32:55<1:30:48, 50.45s/it]
 70%|███████   | 254/361 [3:33:48<1:31:25, 51.26s/it]
                                                     
{'loss': 1.4894, 'learning_rate': 0.00012298850574712642, 'epoch': 0.7}

 70%|███████   | 254/361 [3:33:48<1:31:25, 51.26s/it]
 71%|███████   | 255/361 [3:34:35<1:28:37, 50.17s/it]
                                                     
{'loss': 1.4357, 'learning_rate': 0.0001218390804597701, 'epoch': 0.71}

 71%|███████   | 255/361 [3:34:35<1:28:37, 50.17s/it]
 71%|███████   | 256/361 [3:35:27<1:28:33, 50.61s/it]
                                                     
{'loss': 1.5715, 'learning_rate': 0.00012068965517241378, 'epoch': 0.71}

 71%|███████   | 256/361 [3:35:27<1:28:33, 50.61s/it]
 71%|███████   | 257/361 [3:36:21<1:29:12, 51.47s/it]
                                                     
{'loss': 1.4784, 'learning_rate': 0.00011954022988505745, 'epoch': 0.71}

 71%|███████   | 257/361 [3:36:21<1:29:12, 51.47s/it]
 71%|███████▏  | 258/361 [3:37:12<1:28:06, 51.33s/it]
                                                     
{'loss': 1.4611, 'learning_rate': 0.00011839080459770114, 'epoch': 0.71}

 71%|███████▏  | 258/361 [3:37:12<1:28:06, 51.33s/it]
 72%|███████▏  | 259/361 [3:38:04<1:27:43, 51.60s/it]
                                                     
{'loss': 1.4678, 'learning_rate': 0.00011724137931034482, 'epoch': 0.72}

 72%|███████▏  | 259/361 [3:38:04<1:27:43, 51.60s/it]
 72%|███████▏  | 260/361 [3:38:51<1:24:50, 50.40s/it]
                                                     
{'loss': 1.4347, 'learning_rate': 0.0001160919540229885, 'epoch': 0.72}

 72%|███████▏  | 260/361 [3:38:51<1:24:50, 50.40s/it]
 72%|███████▏  | 261/361 [3:39:39<1:22:23, 49.44s/it]
                                                     
{'loss': 1.5054, 'learning_rate': 0.00011494252873563218, 'epoch': 0.72}

 72%|███████▏  | 261/361 [3:39:39<1:22:23, 49.44s/it]
 73%|███████▎  | 262/361 [3:40:30<1:22:21, 49.91s/it]
                                                     
{'loss': 1.4252, 'learning_rate': 0.00011379310344827585, 'epoch': 0.72}

 73%|███████▎  | 262/361 [3:40:30<1:22:21, 49.91s/it]
 73%|███████▎  | 263/361 [3:41:21<1:22:18, 50.39s/it]
                                                     
{'loss': 1.5133, 'learning_rate': 0.00011264367816091954, 'epoch': 0.73}

 73%|███████▎  | 263/361 [3:41:21<1:22:18, 50.39s/it]
 73%|███████▎  | 264/361 [3:42:12<1:21:46, 50.59s/it]
                                                     
{'loss': 1.4642, 'learning_rate': 0.00011149425287356322, 'epoch': 0.73}

 73%|███████▎  | 264/361 [3:42:12<1:21:46, 50.59s/it]
 73%|███████▎  | 265/361 [3:43:04<1:21:26, 50.90s/it]
                                                     
{'loss': 1.485, 'learning_rate': 0.00011034482758620688, 'epoch': 0.73}

 73%|███████▎  | 265/361 [3:43:04<1:21:26, 50.90s/it]
 74%|███████▎  | 266/361 [3:43:53<1:19:55, 50.48s/it]
                                                     
{'loss': 1.4835, 'learning_rate': 0.00010919540229885056, 'epoch': 0.74}

 74%|███████▎  | 266/361 [3:43:53<1:19:55, 50.48s/it]
 74%|███████▍  | 267/361 [3:44:47<1:20:35, 51.44s/it]
                                                     
{'loss': 1.4749, 'learning_rate': 0.00010804597701149424, 'epoch': 0.74}

 74%|███████▍  | 267/361 [3:44:47<1:20:35, 51.44s/it]
 74%|███████▍  | 268/361 [3:45:35<1:18:07, 50.40s/it]
                                                     
{'loss': 1.5396, 'learning_rate': 0.00010689655172413792, 'epoch': 0.74}

 74%|███████▍  | 268/361 [3:45:35<1:18:07, 50.40s/it]
 75%|███████▍  | 269/361 [3:46:27<1:18:07, 50.95s/it]
                                                     
{'loss': 1.4035, 'learning_rate': 0.00010574712643678159, 'epoch': 0.74}

 75%|███████▍  | 269/361 [3:46:27<1:18:07, 50.95s/it]
 75%|███████▍  | 270/361 [3:47:17<1:16:50, 50.67s/it]
                                                     
{'loss': 1.4964, 'learning_rate': 0.00010459770114942528, 'epoch': 0.75}

 75%|███████▍  | 270/361 [3:47:17<1:16:50, 50.67s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 75%|███████▌  | 271/361 [3:48:09<1:16:26, 50.96s/it]
                                                     
{'loss': 1.5063, 'learning_rate': 0.00010344827586206896, 'epoch': 0.75}

 75%|███████▌  | 271/361 [3:48:09<1:16:26, 50.96s/it]
 75%|███████▌  | 272/361 [3:48:59<1:15:05, 50.62s/it]
                                                     
{'loss': 1.5319, 'learning_rate': 0.00010229885057471263, 'epoch': 0.75}

 75%|███████▌  | 272/361 [3:48:59<1:15:05, 50.62s/it]
 76%|███████▌  | 273/361 [3:49:48<1:13:53, 50.38s/it]
                                                     
{'loss': 1.46, 'learning_rate': 0.00010114942528735632, 'epoch': 0.75}

 76%|███████▌  | 273/361 [3:49:48<1:13:53, 50.38s/it]
 76%|███████▌  | 274/361 [3:50:38<1:12:27, 49.97s/it]
                                                     
{'loss': 1.4678, 'learning_rate': 9.999999999999999e-05, 'epoch': 0.76}

 76%|███████▌  | 274/361 [3:50:38<1:12:27, 49.97s/it]
 76%|███████▌  | 275/361 [3:51:26<1:10:59, 49.53s/it]
                                                     
{'loss': 1.4314, 'learning_rate': 9.885057471264368e-05, 'epoch': 0.76}

 76%|███████▌  | 275/361 [3:51:26<1:10:59, 49.53s/it]
 76%|███████▋  | 276/361 [3:52:12<1:08:39, 48.47s/it]
                                                     
{'loss': 1.4827, 'learning_rate': 9.770114942528733e-05, 'epoch': 0.76}

 76%|███████▋  | 276/361 [3:52:12<1:08:39, 48.47s/it]
 77%|███████▋  | 277/361 [3:53:01<1:07:56, 48.53s/it]
                                                     
{'loss': 1.392, 'learning_rate': 9.655172413793102e-05, 'epoch': 0.77}

 77%|███████▋  | 277/361 [3:53:01<1:07:56, 48.53s/it]
 77%|███████▋  | 278/361 [3:53:49<1:07:06, 48.51s/it]
                                                     
{'loss': 1.454, 'learning_rate': 9.54022988505747e-05, 'epoch': 0.77}

 77%|███████▋  | 278/361 [3:53:49<1:07:06, 48.51s/it]
 77%|███████▋  | 279/361 [3:54:39<1:06:45, 48.85s/it]
                                                     
{'loss': 1.4781, 'learning_rate': 9.425287356321838e-05, 'epoch': 0.77}

 77%|███████▋  | 279/361 [3:54:39<1:06:45, 48.85s/it]
 78%|███████▊  | 280/361 [3:55:25<1:05:00, 48.15s/it]
                                                     
{'loss': 1.4687, 'learning_rate': 9.310344827586206e-05, 'epoch': 0.77}

 78%|███████▊  | 280/361 [3:55:25<1:05:00, 48.15s/it]
 78%|███████▊  | 281/361 [3:56:12<1:03:31, 47.65s/it]
                                                     
{'loss': 1.5677, 'learning_rate': 9.195402298850573e-05, 'epoch': 0.78}

 78%|███████▊  | 281/361 [3:56:12<1:03:31, 47.65s/it]
 78%|███████▊  | 282/361 [3:57:00<1:02:49, 47.72s/it]
                                                     
{'loss': 1.5413, 'learning_rate': 9.080459770114942e-05, 'epoch': 0.78}

 78%|███████▊  | 282/361 [3:57:00<1:02:49, 47.72s/it]
 78%|███████▊  | 283/361 [3:57:49<1:02:42, 48.23s/it]
                                                     
{'loss': 1.4611, 'learning_rate': 8.96551724137931e-05, 'epoch': 0.78}

 78%|███████▊  | 283/361 [3:57:49<1:02:42, 48.23s/it]
 79%|███████▊  | 284/361 [3:58:38<1:02:11, 48.46s/it]
                                                     
{'loss': 1.3973, 'learning_rate': 8.850574712643677e-05, 'epoch': 0.79}

 79%|███████▊  | 284/361 [3:58:38<1:02:11, 48.46s/it]
 79%|███████▉  | 285/361 [3:59:25<1:00:36, 47.85s/it]
                                                     
{'loss': 1.5091, 'learning_rate': 8.735632183908046e-05, 'epoch': 0.79}

 79%|███████▉  | 285/361 [3:59:25<1:00:36, 47.85s/it]
 79%|███████▉  | 286/361 [4:00:13<59:55, 47.94s/it]  
                                                   
{'loss': 1.5234, 'learning_rate': 8.620689655172413e-05, 'epoch': 0.79}

 79%|███████▉  | 286/361 [4:00:13<59:55, 47.94s/it]
 80%|███████▉  | 287/361 [4:01:02<59:49, 48.50s/it]
                                                   
{'loss': 1.4924, 'learning_rate': 8.505747126436782e-05, 'epoch': 0.79}

 80%|███████▉  | 287/361 [4:01:03<59:49, 48.50s/it]
 80%|███████▉  | 288/361 [4:01:51<59:05, 48.57s/it]
                                                   
{'loss': 1.5232, 'learning_rate': 8.390804597701147e-05, 'epoch': 0.8}

 80%|███████▉  | 288/361 [4:01:51<59:05, 48.57s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 80%|████████  | 289/361 [4:02:40<58:23, 48.66s/it]
                                                   
{'loss': 1.4043, 'learning_rate': 8.275862068965516e-05, 'epoch': 0.8}

 80%|████████  | 289/361 [4:02:40<58:23, 48.66s/it]
 80%|████████  | 290/361 [4:03:29<57:43, 48.78s/it]
                                                   
{'loss': 1.5269, 'learning_rate': 8.160919540229884e-05, 'epoch': 0.8}

 80%|████████  | 290/361 [4:03:29<57:43, 48.78s/it]
 81%|████████  | 291/361 [4:04:16<56:22, 48.32s/it]
                                                   
{'loss': 1.5002, 'learning_rate': 8.045977011494252e-05, 'epoch': 0.8}

 81%|████████  | 291/361 [4:04:16<56:22, 48.32s/it]
 81%|████████  | 292/361 [4:05:06<55:53, 48.60s/it]
                                                   
{'loss': 1.4514, 'learning_rate': 7.93103448275862e-05, 'epoch': 0.81}

 81%|████████  | 292/361 [4:05:06<55:53, 48.60s/it]
 81%|████████  | 293/361 [4:05:53<54:46, 48.33s/it]
                                                   
{'loss': 1.4452, 'learning_rate': 7.816091954022987e-05, 'epoch': 0.81}

 81%|████████  | 293/361 [4:05:53<54:46, 48.33s/it]
 81%|████████▏ | 294/361 [4:06:43<54:24, 48.73s/it]
                                                   
{'loss': 1.4641, 'learning_rate': 7.701149425287356e-05, 'epoch': 0.81}

 81%|████████▏ | 294/361 [4:06:43<54:24, 48.73s/it]
 82%|████████▏ | 295/361 [4:07:32<53:37, 48.75s/it]
                                                   
{'loss': 1.4966, 'learning_rate': 7.586206896551724e-05, 'epoch': 0.82}

 82%|████████▏ | 295/361 [4:07:32<53:37, 48.75s/it]
 82%|████████▏ | 296/361 [4:08:24<53:49, 49.68s/it]
                                                   
{'loss': 1.4361, 'learning_rate': 7.471264367816091e-05, 'epoch': 0.82}

 82%|████████▏ | 296/361 [4:08:24<53:49, 49.68s/it]
 82%|████████▏ | 297/361 [4:09:13<52:48, 49.51s/it]
                                                   
{'loss': 1.4633, 'learning_rate': 7.356321839080459e-05, 'epoch': 0.82}

 82%|████████▏ | 297/361 [4:09:13<52:48, 49.51s/it]
 83%|████████▎ | 298/361 [4:09:59<50:57, 48.54s/it]
                                                   
{'loss': 1.4884, 'learning_rate': 7.241379310344827e-05, 'epoch': 0.82}

 83%|████████▎ | 298/361 [4:09:59<50:57, 48.54s/it]
 83%|████████▎ | 299/361 [4:10:47<49:59, 48.38s/it]
                                                   
{'loss': 1.4783, 'learning_rate': 7.126436781609194e-05, 'epoch': 0.83}

 83%|████████▎ | 299/361 [4:10:47<49:59, 48.38s/it]
 83%|████████▎ | 300/361 [4:11:37<49:32, 48.73s/it]
                                                   
{'loss': 1.4627, 'learning_rate': 7.011494252873563e-05, 'epoch': 0.83}

 83%|████████▎ | 300/361 [4:11:37<49:32, 48.73s/it]
 83%|████████▎ | 301/361 [4:12:29<49:42, 49.70s/it]
                                                   
{'loss': 1.438, 'learning_rate': 6.89655172413793e-05, 'epoch': 0.83}

 83%|████████▎ | 301/361 [4:12:29<49:42, 49.70s/it]
 84%|████████▎ | 302/361 [4:13:17<48:24, 49.24s/it]
                                                   
{'loss': 1.4409, 'learning_rate': 6.781609195402298e-05, 'epoch': 0.84}

 84%|████████▎ | 302/361 [4:13:17<48:24, 49.24s/it]
 84%|████████▍ | 303/361 [4:14:08<48:12, 49.86s/it]
                                                   
{'loss': 1.4435, 'learning_rate': 6.666666666666666e-05, 'epoch': 0.84}

 84%|████████▍ | 303/361 [4:14:08<48:12, 49.86s/it]
 84%|████████▍ | 304/361 [4:14:59<47:37, 50.13s/it]
                                                   
{'loss': 1.5322, 'learning_rate': 6.551724137931034e-05, 'epoch': 0.84}

 84%|████████▍ | 304/361 [4:14:59<47:37, 50.13s/it]
 84%|████████▍ | 305/361 [4:15:48<46:38, 49.97s/it]
                                                   
{'loss': 1.5397, 'learning_rate': 6.436781609195401e-05, 'epoch': 0.84}

 84%|████████▍ | 305/361 [4:15:48<46:38, 49.97s/it]
 85%|████████▍ | 306/361 [4:16:37<45:26, 49.57s/it]
                                                   
{'loss': 1.4385, 'learning_rate': 6.32183908045977e-05, 'epoch': 0.85}

 85%|████████▍ | 306/361 [4:16:37<45:26, 49.57s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 85%|████████▌ | 307/361 [4:17:27<44:43, 49.69s/it]
                                                   
{'loss': 1.4512, 'learning_rate': 6.206896551724137e-05, 'epoch': 0.85}

 85%|████████▌ | 307/361 [4:17:27<44:43, 49.69s/it]
 85%|████████▌ | 308/361 [4:18:16<43:48, 49.59s/it]
                                                   
{'loss': 1.3932, 'learning_rate': 6.091954022988505e-05, 'epoch': 0.85}

 85%|████████▌ | 308/361 [4:18:16<43:48, 49.59s/it]
 86%|████████▌ | 309/361 [4:19:05<42:40, 49.24s/it]
                                                   
{'loss': 1.5121, 'learning_rate': 5.9770114942528726e-05, 'epoch': 0.85}

 86%|████████▌ | 309/361 [4:19:05<42:40, 49.24s/it]
 86%|████████▌ | 310/361 [4:19:52<41:19, 48.62s/it]
                                                   
{'loss': 1.4713, 'learning_rate': 5.862068965517241e-05, 'epoch': 0.86}

 86%|████████▌ | 310/361 [4:19:52<41:19, 48.62s/it]
 86%|████████▌ | 311/361 [4:20:41<40:37, 48.75s/it]
                                                   
{'loss': 1.4904, 'learning_rate': 5.747126436781609e-05, 'epoch': 0.86}

 86%|████████▌ | 311/361 [4:20:41<40:37, 48.75s/it]
 86%|████████▋ | 312/361 [4:21:32<40:17, 49.33s/it]
                                                   
{'loss': 1.4974, 'learning_rate': 5.632183908045977e-05, 'epoch': 0.86}

 86%|████████▋ | 312/361 [4:21:32<40:17, 49.33s/it]
 87%|████████▋ | 313/361 [4:22:20<39:11, 48.99s/it]
                                                   
{'loss': 1.5157, 'learning_rate': 5.517241379310344e-05, 'epoch': 0.87}

 87%|████████▋ | 313/361 [4:22:20<39:11, 48.99s/it]
 87%|████████▋ | 314/361 [4:23:06<37:48, 48.28s/it]
                                                   
{'loss': 1.4576, 'learning_rate': 5.402298850574712e-05, 'epoch': 0.87}

 87%|████████▋ | 314/361 [4:23:06<37:48, 48.28s/it]
 87%|████████▋ | 315/361 [4:23:56<37:12, 48.53s/it]
                                                   
{'loss': 1.514, 'learning_rate': 5.2873563218390796e-05, 'epoch': 0.87}

 87%|████████▋ | 315/361 [4:23:56<37:12, 48.53s/it]
 88%|████████▊ | 316/361 [4:24:44<36:28, 48.63s/it]
                                                   
{'loss': 1.4151, 'learning_rate': 5.172413793103448e-05, 'epoch': 0.87}

 88%|████████▊ | 316/361 [4:24:44<36:28, 48.63s/it]
 88%|████████▊ | 317/361 [4:25:34<35:48, 48.83s/it]
                                                   
{'loss': 1.3763, 'learning_rate': 5.057471264367816e-05, 'epoch': 0.88}

 88%|████████▊ | 317/361 [4:25:34<35:48, 48.83s/it]
 88%|████████▊ | 318/361 [4:26:25<35:27, 49.48s/it]
                                                   
{'loss': 1.451, 'learning_rate': 4.942528735632184e-05, 'epoch': 0.88}

 88%|████████▊ | 318/361 [4:26:25<35:27, 49.48s/it]
 88%|████████▊ | 319/361 [4:27:13<34:21, 49.08s/it]
                                                   
{'loss': 1.4855, 'learning_rate': 4.827586206896551e-05, 'epoch': 0.88}

 88%|████████▊ | 319/361 [4:27:13<34:21, 49.08s/it]
 89%|████████▊ | 320/361 [4:28:00<33:07, 48.48s/it]
                                                   
{'loss': 1.5891, 'learning_rate': 4.712643678160919e-05, 'epoch': 0.88}

 89%|████████▊ | 320/361 [4:28:00<33:07, 48.48s/it]
 89%|████████▉ | 321/361 [4:28:50<32:32, 48.81s/it]
                                                   
{'loss': 1.3762, 'learning_rate': 4.5977011494252866e-05, 'epoch': 0.89}

 89%|████████▉ | 321/361 [4:28:50<32:32, 48.81s/it]
 89%|████████▉ | 322/361 [4:29:38<31:42, 48.77s/it]
                                                   
{'loss': 1.4777, 'learning_rate': 4.482758620689655e-05, 'epoch': 0.89}

 89%|████████▉ | 322/361 [4:29:38<31:42, 48.77s/it]
 89%|████████▉ | 323/361 [4:30:26<30:40, 48.42s/it]
                                                   
{'loss': 1.5266, 'learning_rate': 4.367816091954023e-05, 'epoch': 0.89}

 89%|████████▉ | 323/361 [4:30:26<30:40, 48.42s/it]
 90%|████████▉ | 324/361 [4:31:18<30:29, 49.44s/it]
                                                   
{'loss': 1.5228, 'learning_rate': 4.252873563218391e-05, 'epoch': 0.9}

 90%|████████▉ | 324/361 [4:31:18<30:29, 49.44s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 90%|█████████ | 325/361 [4:32:07<29:38, 49.40s/it]
                                                   
{'loss': 1.4449, 'learning_rate': 4.137931034482758e-05, 'epoch': 0.9}

 90%|█████████ | 325/361 [4:32:07<29:38, 49.40s/it]
 90%|█████████ | 326/361 [4:32:55<28:30, 48.86s/it]
                                                   
{'loss': 1.5498, 'learning_rate': 4.022988505747126e-05, 'epoch': 0.9}

 90%|█████████ | 326/361 [4:32:55<28:30, 48.86s/it]
 91%|█████████ | 327/361 [4:33:45<27:58, 49.37s/it]
                                                   
{'loss': 1.4187, 'learning_rate': 3.9080459770114936e-05, 'epoch': 0.9}

 91%|█████████ | 327/361 [4:33:45<27:58, 49.37s/it]
 91%|█████████ | 328/361 [4:34:32<26:48, 48.75s/it]
                                                   
{'loss': 1.4098, 'learning_rate': 3.793103448275862e-05, 'epoch': 0.91}

 91%|█████████ | 328/361 [4:34:32<26:48, 48.75s/it]
 91%|█████████ | 329/361 [4:35:19<25:36, 48.03s/it]
                                                   
{'loss': 1.5546, 'learning_rate': 3.678160919540229e-05, 'epoch': 0.91}

 91%|█████████ | 329/361 [4:35:19<25:36, 48.03s/it]
 91%|█████████▏| 330/361 [4:36:11<25:27, 49.27s/it]
                                                   
{'loss': 1.4452, 'learning_rate': 3.563218390804597e-05, 'epoch': 0.91}

 91%|█████████▏| 330/361 [4:36:11<25:27, 49.27s/it]
 92%|█████████▏| 331/361 [4:37:02<24:51, 49.71s/it]
                                                   
{'loss': 1.3997, 'learning_rate': 3.448275862068965e-05, 'epoch': 0.92}

 92%|█████████▏| 331/361 [4:37:02<24:51, 49.71s/it]
 92%|█████████▏| 332/361 [4:37:50<23:47, 49.21s/it]
                                                   
{'loss': 1.4311, 'learning_rate': 3.333333333333333e-05, 'epoch': 0.92}

 92%|█████████▏| 332/361 [4:37:50<23:47, 49.21s/it]
 92%|█████████▏| 333/361 [4:38:38<22:52, 49.03s/it]
                                                   
{'loss': 1.3731, 'learning_rate': 3.2183908045977006e-05, 'epoch': 0.92}

 92%|█████████▏| 333/361 [4:38:38<22:52, 49.03s/it]
 93%|█████████▎| 334/361 [4:39:28<22:06, 49.14s/it]
                                                   
{'loss': 1.4261, 'learning_rate': 3.1034482758620685e-05, 'epoch': 0.92}

 93%|█████████▎| 334/361 [4:39:28<22:06, 49.14s/it]
 93%|█████████▎| 335/361 [4:40:18<21:22, 49.34s/it]
                                                   
{'loss': 1.5071, 'learning_rate': 2.9885057471264363e-05, 'epoch': 0.93}

 93%|█████████▎| 335/361 [4:40:18<21:22, 49.34s/it]
 93%|█████████▎| 336/361 [4:41:06<20:25, 49.01s/it]
                                                   
{'loss': 1.4374, 'learning_rate': 2.8735632183908045e-05, 'epoch': 0.93}

 93%|█████████▎| 336/361 [4:41:06<20:25, 49.01s/it]
 93%|█████████▎| 337/361 [4:41:55<19:34, 48.93s/it]
                                                   
{'loss': 1.4424, 'learning_rate': 2.758620689655172e-05, 'epoch': 0.93}

 93%|█████████▎| 337/361 [4:41:55<19:34, 48.93s/it]
 94%|█████████▎| 338/361 [4:42:42<18:38, 48.64s/it]
                                                   
{'loss': 1.5217, 'learning_rate': 2.6436781609195398e-05, 'epoch': 0.93}

 94%|█████████▎| 338/361 [4:42:42<18:38, 48.64s/it]
 94%|█████████▍| 339/361 [4:43:33<18:00, 49.09s/it]
                                                   
{'loss': 1.4083, 'learning_rate': 2.528735632183908e-05, 'epoch': 0.94}

 94%|█████████▍| 339/361 [4:43:33<18:00, 49.09s/it]
 94%|█████████▍| 340/361 [4:44:20<16:59, 48.55s/it]
                                                   
{'loss': 1.6009, 'learning_rate': 2.4137931034482755e-05, 'epoch': 0.94}

 94%|█████████▍| 340/361 [4:44:20<16:59, 48.55s/it]
 94%|█████████▍| 341/361 [4:45:07<16:03, 48.18s/it]
                                                   
{'loss': 1.4725, 'learning_rate': 2.2988505747126433e-05, 'epoch': 0.94}

 94%|█████████▍| 341/361 [4:45:07<16:03, 48.18s/it]
 95%|█████████▍| 342/361 [4:45:56<15:16, 48.22s/it]
                                                   
{'loss': 1.4918, 'learning_rate': 2.1839080459770115e-05, 'epoch': 0.95}

 95%|█████████▍| 342/361 [4:45:56<15:16, 48.22s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 95%|█████████▌| 343/361 [4:46:46<14:38, 48.78s/it]
                                                   
{'loss': 1.4974, 'learning_rate': 2.068965517241379e-05, 'epoch': 0.95}

 95%|█████████▌| 343/361 [4:46:46<14:38, 48.78s/it]
 95%|█████████▌| 344/361 [4:47:35<13:50, 48.83s/it]
                                                   
{'loss': 1.49, 'learning_rate': 1.9540229885057468e-05, 'epoch': 0.95}

 95%|█████████▌| 344/361 [4:47:35<13:50, 48.83s/it]
 96%|█████████▌| 345/361 [4:48:22<12:55, 48.46s/it]
                                                   
{'loss': 1.4797, 'learning_rate': 1.8390804597701147e-05, 'epoch': 0.95}

 96%|█████████▌| 345/361 [4:48:22<12:55, 48.46s/it]
 96%|█████████▌| 346/361 [4:49:09<11:57, 47.86s/it]
                                                   
{'loss': 1.4433, 'learning_rate': 1.7241379310344825e-05, 'epoch': 0.96}

 96%|█████████▌| 346/361 [4:49:09<11:57, 47.86s/it]
 96%|█████████▌| 347/361 [4:49:56<11:07, 47.71s/it]
                                                   
{'loss': 1.399, 'learning_rate': 1.6091954022988503e-05, 'epoch': 0.96}

 96%|█████████▌| 347/361 [4:49:56<11:07, 47.71s/it]
 96%|█████████▋| 348/361 [4:50:46<10:29, 48.44s/it]
                                                   
{'loss': 1.4508, 'learning_rate': 1.4942528735632182e-05, 'epoch': 0.96}

 96%|█████████▋| 348/361 [4:50:46<10:29, 48.44s/it]
 97%|█████████▋| 349/361 [4:51:34<09:39, 48.25s/it]
                                                   
{'loss': 1.4699, 'learning_rate': 1.379310344827586e-05, 'epoch': 0.97}

 97%|█████████▋| 349/361 [4:51:34<09:39, 48.25s/it]
 97%|█████████▋| 350/361 [4:52:23<08:52, 48.36s/it]
                                                   
{'loss': 1.4684, 'learning_rate': 1.264367816091954e-05, 'epoch': 0.97}

 97%|█████████▋| 350/361 [4:52:23<08:52, 48.36s/it]
 97%|█████████▋| 351/361 [4:53:11<08:04, 48.42s/it]
                                                   
{'loss': 1.4643, 'learning_rate': 1.1494252873563217e-05, 'epoch': 0.97}

 97%|█████████▋| 351/361 [4:53:11<08:04, 48.42s/it]
 98%|█████████▊| 352/361 [4:53:59<07:14, 48.28s/it]
                                                   
{'loss': 1.4383, 'learning_rate': 1.0344827586206895e-05, 'epoch': 0.97}

 98%|█████████▊| 352/361 [4:53:59<07:14, 48.28s/it]
 98%|█████████▊| 353/361 [4:54:48<06:26, 48.35s/it]
                                                   
{'loss': 1.4632, 'learning_rate': 9.195402298850573e-06, 'epoch': 0.98}

 98%|█████████▊| 353/361 [4:54:48<06:26, 48.35s/it]
 98%|█████████▊| 354/361 [4:55:38<05:42, 48.90s/it]
                                                   
{'loss': 1.4312, 'learning_rate': 8.045977011494252e-06, 'epoch': 0.98}

 98%|█████████▊| 354/361 [4:55:38<05:42, 48.90s/it]
 98%|█████████▊| 355/361 [4:56:28<04:55, 49.32s/it]
                                                   
{'loss': 1.4374, 'learning_rate': 6.89655172413793e-06, 'epoch': 0.98}

 98%|█████████▊| 355/361 [4:56:28<04:55, 49.32s/it]
 99%|█████████▊| 356/361 [4:57:16<04:04, 48.89s/it]
                                                   
{'loss': 1.3916, 'learning_rate': 5.747126436781608e-06, 'epoch': 0.98}

 99%|█████████▊| 356/361 [4:57:16<04:04, 48.89s/it]
 99%|█████████▉| 357/361 [4:58:05<03:15, 48.98s/it]
                                                   
{'loss': 1.4845, 'learning_rate': 4.597701149425287e-06, 'epoch': 0.99}

 99%|█████████▉| 357/361 [4:58:05<03:15, 48.98s/it]
 99%|█████████▉| 358/361 [4:58:55<02:27, 49.22s/it]
                                                   
{'loss': 1.3987, 'learning_rate': 3.448275862068965e-06, 'epoch': 0.99}

 99%|█████████▉| 358/361 [4:58:55<02:27, 49.22s/it]
 99%|█████████▉| 359/361 [4:59:43<01:37, 48.79s/it]
                                                   
{'loss': 1.4713, 'learning_rate': 2.2988505747126433e-06, 'epoch': 0.99}

 99%|█████████▉| 359/361 [4:59:43<01:37, 48.79s/it]
100%|█████████▉| 360/361 [5:00:32<00:48, 48.93s/it]
                                                   
{'loss': 1.4754, 'learning_rate': 1.1494252873563217e-06, 'epoch': 1.0}

100%|█████████▉| 360/361 [5:00:32<00:48, 48.93s/it]/nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

100%|██████████| 361/361 [5:01:23<00:00, 49.60s/it]
                                                   
{'loss': 1.3954, 'learning_rate': 0.0, 'epoch': 1.0}

100%|██████████| 361/361 [5:01:23<00:00, 49.60s/it]
                                                   
{'train_runtime': 18085.6927, 'train_samples_per_second': 2.559, 'train_steps_per_second': 0.02, 'train_loss': 1.5041567341442583, 'epoch': 1.0}

100%|██████████| 361/361 [5:01:23<00:00, 49.60s/it]
100%|██████████| 361/361 [5:01:23<00:00, 50.09s/it]

 If there's a warning about missing keys above, please disregard :)
