WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/ubuntu/miniconda3/envs/h2ollm did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so...
bin /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so
bin /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so
bin /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so
bin/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/ubuntu/miniconda3/envs/h2ollm did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
 CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so

CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so...
bin /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so
bin /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so
bin /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/ubuntu/miniconda3/envs/h2ollm did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so...
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/ubuntu/miniconda3/envs/h2ollm did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so...
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/ubuntu/miniconda3/envs/h2ollm did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so...
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/ubuntu/miniconda3/envs/h2ollm did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so...
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/ubuntu/miniconda3/envs/h2ollm did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so...
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/ubuntu/miniconda3/envs/h2ollm did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so...
local_rank: 7
global rank: 7
local_rank: 6
global rank: 6
local_rank: 1
global rank: 1
local_rank: 3
global rank: 3
local_rank: 4
global rank: 4
local_rank: 2
global rank: 2
local_rank: 5
global rank: 5
No neptune configured, set NEPTUNE_API_TOKEN env var.

    Example runs on 4 GPUs:
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='decapoda-research/llama-7b-hf' --data_path=data/config.json --run_id=0 &> 0.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='decapoda-research/llama-30b-hf' --data_path=data/config.json --batch_size=16 --micro_batch_size=1 --run_id=1 --save_code=True &> 1.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='EleutherAI/gpt-j-6B' --data_path=data/config.json --run_id=2 &> 2.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='EleutherAI/gpt-neox-20b' --data_path=data/config.json --run_id=8 --batch_size=16 --micro_batch_size=4 &> 8.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='togethercomputer/GPT-NeoXT-Chat-Base-20B' --data_path=data/config.json --prompt_type='dai_faq' --run_id=13 --batch_size=16 --micro_batch_size=4 --num_epochs=100 --val_set_size=0 data_mix_in_path='' &> 13.log
    WORLD_SIZE=4 CUDA_VISIBLE_DEVICES="0,1,2,3" torchrun --nproc_per_node=4 finetune.py --base_model='togethercomputer/GPT-NeoXT-Chat-Base-20B' --data_path=data/config.json --run_id=28 --batch_size=16 --micro_batch_size=4 --num_epochs=8 --val_set_size=0 --data_mix_in_factor=0.1 --data_mix_in_prompt_type='human_bot' --save_code=True --cutoff_len=512  &> 28.log

    All metrics:
    CUDA_VISIBLE_DEVICES= finetune.py --data_mix_in_factor=0 --eval_steps=100 --warmup_steps=2 --val_set_size=100 --val_metrics="['bleu', 'rouge', 'sacrebleu', 'meteor']"

    # Fine-tune 20B on 24GB GPUs across 3 nodes with 3+2+2 GPUs
    rippa>
NCCL_P2P_LEVEL=LOC WORLD_SIZE=7 CUDA_VISIBLE_DEVICES="0,1,2" torchrun --node_rank 0 --nproc_per_node=3 --master_port=1234 --nnodes=3 --master_addr=10.10.10.2 finetune.py --data_path=merged_shuffled_OIG_87f6a1e788.json --micro_batch_size=1 --batch_size=7 --cutoff_len=512 --run_id=17 &>log.17.rank0
    ova>
NCCL_P2P_LEVEL=LOC WORLD_SIZE=7 CUDA_VISIBLE_DEVICES="0,1" torchrun --node_rank 1 --nproc_per_node=2 --master_port=1234 --nnodes=3 --master_addr=10.10.10.2 finetune.py --data_path=merged_shuffled_OIG_87f6a1e788.json --micro_batch_size=1 --batch_size=7 --cutoff_len=512 --run_id=17 &>log.17.rank1
    timemachine>
NCCL_P2P_LEVEL=LOC WORLD_SIZE=7 CUDA_VISIBLE_DEVICES="0,1" torchrun --node_rank 2 --nproc_per_node=2 --master_port=1234 --nnodes=3 --master_addr=10.10.10.2 finetune.py --data_path=merged_shuffled_OIG_87f6a1e788.json --micro_batch_size=1 --batch_size=7 --cutoff_len=512 --run_id=17 &>log.17.rank2

    
local_rank: 0
global rank: 0
Training model with params:
save_code: False
run_id: 16
base_model: EleutherAI/pythia-12b-deduped
tokenizer_base_model: EleutherAI/pythia-12b-deduped
data_path: openassistant_oasst1.json
data_col_dict: None
valid_path: None
data_mix_in_path: 0-hero/OIG-small-chip2
data_mix_in_factor: 0.0
data_mix_in_col_dict: {'user': 'instruction', 'chip2': 'output'}
data_mix_in_prompt_type: instruct
output_dir: pythia-12b-deduped.openassistant_oasst1.json.1_epochs.5fc91911bc2bfaaf3b6c2de577c4b0ae45a07a4a.16
lora_weights: pythia-12b-deduped.h2ogpt-oig-instruct-cleaned-v2.json.1_epochs.5fc91911bc2bfaaf3b6c2de577c4b0ae45a07a4a.8
batch_size: 512
micro_batch_size: 4
gradient_checkpointing: False
fp16: True
num_epochs: 1
learning_rate: 0.0003
val_set_size: None
val_metrics: []
eval_steps: None
eval_epochs: None
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: None
llama_type: False
group_by_length: False
resume_from_checkpoint: None
ddp: True
local_files_only: False
resume_download: True
warmup_steps: 100
logging_steps: 1
save_steps: None
add_eos_token: False
world_size: 8
local_rank: 0
rank: 0
gpus: 8
device_map: auto
gradient_accumulation_steps: 128
cutoff_len: 2048
prompt_type: plain
train_on_inputs: True
Command: finetune.py --base_model=EleutherAI/pythia-12b-deduped --data_path=openassistant_oasst1.json --lora_weights=pythia-12b-deduped.h2ogpt-oig-instruct-cleaned-v2.json.1_epochs.5fc91911bc2bfaaf3b6c2de577c4b0ae45a07a4a.8 --prompt_type=plain --run_id=16 --micro_batch_size=4 --batch_size=512 --cutoff_len=2048
Hash: 5fc91911bc2bfaaf3b6c2de577c4b0ae45a07a4a
Distributed: data parallel

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:12<00:25, 12.51s/it]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:12<00:25, 12.91s/it]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:13<00:26, 13.10s/it]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:15<00:30, 15.23s/it]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:15<00:30, 15.48s/it]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:15<00:31, 15.64s/it]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:15<00:31, 15.81s/it]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:15<00:31, 15.83s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:22<00:11, 11.12s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:25<00:12, 12.47s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:26<00:13, 13.32s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:27<00:00,  8.29s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:27<00:00,  9.19s/it]

Loading checkpoint shards:  67%|██████▋   | 2/3 [00:28<00:14, 14.16s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:29<00:14, 14.38s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:29<00:14, 14.62s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:30<00:15, 15.00s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:30<00:15, 15.06s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:31<00:00,  9.65s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:31<00:00, 10.62s/it]

Loading checkpoint shards: 100%|██████████| 3/3 [00:32<00:00,  9.98s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:32<00:00, 10.70s/it]

Loading checkpoint shards: 100%|██████████| 3/3 [00:33<00:00,  9.94s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:33<00:00, 11.23s/it]

Loading checkpoint shards: 100%|██████████| 3/3 [00:33<00:00, 10.03s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:33<00:00, 11.28s/it]

Loading checkpoint shards: 100%|██████████| 3/3 [00:34<00:00, 10.01s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:34<00:00, 11.38s/it]

Loading checkpoint shards: 100%|██████████| 3/3 [00:35<00:00, 10.34s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:35<00:00, 11.68s/it]

Loading checkpoint shards: 100%|██████████| 3/3 [00:35<00:00, 10.66s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:35<00:00, 11.91s/it]
PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)
trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013
Found cached dataset json (/home/ubuntu/.cache/huggingface/datasets/json/default-89b8d61a87d07ca5/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 500.99it/s]

Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=16):   0%|          | 33/46283 [00:00<08:27, 91.12 examples/s]
Map (num_proc=16):   0%|          | 147/46283 [00:00<01:59, 387.34 examples/s]
Map (num_proc=16):   1%|          | 543/46283 [00:00<00:32, 1392.16 examples/s]
Map (num_proc=16):   3%|▎         | 1507/46283 [00:00<00:11, 3741.72 examples/s]
Map (num_proc=16):   6%|▌         | 2699/46283 [00:00<00:07, 6091.83 examples/s]
Map (num_proc=16):   9%|▊         | 4048/46283 [00:00<00:05, 8236.45 examples/s]
Map (num_proc=16):  12%|█▏        | 5393/46283 [00:00<00:04, 9757.65 examples/s]
Map (num_proc=16):  15%|█▍        | 6715/46283 [00:01<00:03, 10755.06 examples/s]
Map (num_proc=16):  17%|█▋        | 8096/46283 [00:01<00:03, 11626.78 examples/s]
Map (num_proc=16):  21%|██        | 9499/46283 [00:01<00:02, 12284.73 examples/s]
Map (num_proc=16):  23%|██▎       | 10861/46283 [00:01<00:02, 12622.21 examples/s]
Map (num_proc=16):  26%|██▋       | 12254/46283 [00:01<00:02, 12962.44 examples/s]
Map (num_proc=16):  29%|██▉       | 13584/46283 [00:01<00:02, 13024.00 examples/s]
Map (num_proc=16):  32%|███▏      | 14923/46283 [00:01<00:02, 11811.98 examples/s]
Map (num_proc=16):  35%|███▍      | 16141/46283 [00:01<00:02, 11266.86 examples/s]
Map (num_proc=16):  37%|███▋      | 17319/46283 [00:01<00:02, 10488.32 examples/s]
Map (num_proc=16):  40%|████      | 18569/46283 [00:02<00:02, 11009.20 examples/s]
Map (num_proc=16):  43%|████▎     | 19952/46283 [00:02<00:02, 11748.65 examples/s]
Map (num_proc=16):  46%|████▌     | 21287/46283 [00:02<00:02, 12188.77 examples/s]
Map (num_proc=16):  49%|████▉     | 22633/46283 [00:02<00:01, 12540.67 examples/s]
Map (num_proc=16):  52%|█████▏    | 24052/46283 [00:02<00:01, 12994.04 examples/s]
Map (num_proc=16):  55%|█████▍    | 25422/46283 [00:02<00:01, 13140.03 examples/s]
Map (num_proc=16):  58%|█████▊    | 26840/46283 [00:02<00:01, 13424.37 examples/s]
Map (num_proc=16):  61%|██████    | 28250/46283 [00:02<00:01, 13526.88 examples/s]
Map (num_proc=16):  64%|██████▍   | 29627/46283 [00:02<00:01, 13485.28 examples/s]
Map (num_proc=16):  67%|██████▋   | 31002/46283 [00:03<00:01, 11897.72 examples/s]
Map (num_proc=16):  70%|██████▉   | 32234/46283 [00:03<00:01, 11017.88 examples/s]
Map (num_proc=16):  72%|███████▏  | 33381/46283 [00:03<00:01, 11118.79 examples/s]
Map (num_proc=16):  75%|███████▍  | 34624/46283 [00:03<00:01, 11443.39 examples/s]
Map (num_proc=16):  78%|███████▊  | 35934/46283 [00:03<00:00, 11903.92 examples/s]
Map (num_proc=16):  81%|████████  | 37357/46283 [00:03<00:00, 12539.71 examples/s]
Map (num_proc=16):  84%|████████▎ | 38708/46283 [00:03<00:00, 12800.09 examples/s]
Map (num_proc=16):  87%|████████▋ | 40061/46283 [00:03<00:00, 13000.57 examples/s]
Map (num_proc=16):  90%|████████▉ | 41432/46283 [00:03<00:00, 13194.63 examples/s]
Map (num_proc=16):  93%|█████████▎| 42873/46283 [00:03<00:00, 13547.45 examples/s]
Map (num_proc=16):  96%|█████████▌| 44238/46283 [00:04<00:00, 12651.74 examples/s]
Map (num_proc=16):  98%|█████████▊| 45544/46283 [00:04<00:00, 9797.69 examples/s] PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)
trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013
Found cached dataset json (/home/ubuntu/.cache/huggingface/datasets/json/default-89b8d61a87d07ca5/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 373.49it/s]

                                                                                 
PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)
trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013
Found cached dataset json (/home/ubuntu/.cache/huggingface/datasets/json/default-89b8d61a87d07ca5/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 496.78it/s]
PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)
trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013
PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)
trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013

Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)
trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013

Map (num_proc=16):   0%|          | 55/46283 [00:00<04:38, 165.79 examples/s]Found cached dataset json (/home/ubuntu/.cache/huggingface/datasets/json/default-89b8d61a87d07ca5/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 472.44it/s]

Map (num_proc=16):   1%|          | 251/46283 [00:00<01:06, 696.36 examples/s]PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)

Map (num_proc=16):   2%|▏         | 768/46283 [00:00<00:22, 1980.00 examples/s]trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013
Using Validation Metrics: []
Supported Metrics: ['bleu', 'rouge', 'sacrebleu', 'meteor']
Auto set val_set_size 1000
Found cached dataset json (/home/ubuntu/.cache/huggingface/datasets/json/default-89b8d61a87d07ca5/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 452.66it/s]

Map (num_proc=16):   4%|▎         | 1629/46283 [00:00<00:11, 3830.40 examples/s]
Map (num_proc=16):   5%|▌         | 2383/46283 [00:00<00:08, 4886.34 examples/s]Found cached dataset json (/home/ubuntu/.cache/huggingface/datasets/json/default-89b8d61a87d07ca5/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 507.60it/s]

Map (num_proc=16):   7%|▋         | 3232/46283 [00:00<00:07, 5919.34 examples/s]
Map (num_proc=16):   9%|▉         | 4318/46283 [00:00<00:05, 7342.95 examples/s]
Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=16):  12%|█▏        | 5683/46283 [00:01<00:04, 9166.17 examples/s]Found cached dataset json (/home/ubuntu/.cache/huggingface/datasets/json/default-89b8d61a87d07ca5/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 446.68it/s]

Map (num_proc=16):  15%|█▌        | 7021/46283 [00:01<00:03, 10297.08 examples/s]PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): GPTNeoXForCausalLM(
      (gpt_neox): GPTNeoXModel(
        (embed_in): Embedding(50688, 5120)
        (layers): ModuleList(
          (0-35): 36 x GPTNeoXLayer(
            (input_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (post_attention_layernorm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
            (attention): GPTNeoXAttention(
              (rotary_emb): RotaryEmbedding()
              (query_key_value): MergedLinear8bitLt(
                in_features=5120, out_features=15360, bias=True
                (lora_dropout): Dropout(p=0.05, inplace=False)
                (lora_A): Linear(in_features=5120, out_features=16, bias=False)
                (lora_B): Conv1d(16, 10240, kernel_size=(1,), stride=(1,), groups=2, bias=False)
              )
              (dense): Linear8bitLt(in_features=5120, out_features=5120, bias=True)
            )
            (mlp): GPTNeoXMLP(
              (dense_h_to_4h): Linear8bitLt(in_features=5120, out_features=20480, bias=True)
              (dense_4h_to_h): Linear8bitLt(in_features=20480, out_features=5120, bias=True)
              (act): GELUActivation()
            )
          )
        )
        (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      )
      (embed_out): Linear(in_features=5120, out_features=50688, bias=False)
    )
  )
)
trainable params: 5898240 || all params: 11851970560 || trainable%: 0.04976590154473013

Map (num_proc=16):  18%|█▊        | 8337/46283 [00:01<00:03, 11036.25 examples/s]
Map (num_proc=16):   0%|          | 27/46283 [00:00<08:23, 91.82 examples/s]
Map (num_proc=16):  21%|██        | 9721/46283 [00:01<00:03, 11828.42 examples/s]
Map (num_proc=16):   1%|          | 307/46283 [00:00<00:49, 935.98 examples/s]
Map (num_proc=16):  24%|██▎       | 10956/46283 [00:01<00:02, 11896.20 examples/s]
Map (num_proc=16):   2%|▏         | 930/46283 [00:00<00:19, 2365.93 examples/s]
Map (num_proc=16):  26%|██▋       | 12220/46283 [00:01<00:02, 12092.42 examples/s]
Map (num_proc=16):   4%|▍         | 1748/46283 [00:00<00:11, 4036.30 examples/s]
Map (num_proc=16):  29%|██▉       | 13460/46283 [00:01<00:02, 12133.95 examples/s]
Map (num_proc=16):   5%|▌         | 2454/46283 [00:00<00:08, 4903.64 examples/s]
Map (num_proc=16):  32%|███▏      | 14700/46283 [00:01<00:02, 11354.31 examples/s]
Map (num_proc=16):   7%|▋         | 3332/46283 [00:00<00:07, 5854.57 examples/s]Found cached dataset json (/home/ubuntu/.cache/huggingface/datasets/json/default-89b8d61a87d07ca5/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 442.58it/s]

Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=16):  35%|███▍      | 15986/46283 [00:01<00:02, 11561.80 examples/s]
Map (num_proc=16):  10%|▉         | 4513/46283 [00:00<00:05, 7320.62 examples/s]
Map (num_proc=16):  37%|███▋      | 17331/46283 [00:02<00:02, 12074.44 examples/s]
Map (num_proc=16):  13%|█▎        | 5871/46283 [00:01<00:04, 9054.15 examples/s]
Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=16):  40%|████      | 18569/46283 [00:02<00:02, 11333.03 examples/s]
Map (num_proc=16):  16%|█▌        | 7175/46283 [00:01<00:03, 10138.81 examples/s]
Map (num_proc=16):   0%|          | 35/46283 [00:00<06:09, 125.09 examples/s]
Map (num_proc=16):  18%|█▊        | 8494/46283 [00:01<00:03, 11016.93 examples/s]
Map (num_proc=16):  43%|████▎     | 19751/46283 [00:02<00:02, 11176.58 examples/s]
Map (num_proc=16):   0%|          | 191/46283 [00:00<01:18, 585.26 examples/s]
Map (num_proc=16):  21%|██        | 9818/46283 [00:01<00:03, 11632.07 examples/s]
Map (num_proc=16):  46%|████▌     | 21080/46283 [00:02<00:02, 11738.90 examples/s]
Map (num_proc=16):   2%|▏         | 778/46283 [00:00<00:21, 2131.56 examples/s]
Map (num_proc=16):   0%|          | 33/46283 [00:00<07:33, 102.08 examples/s]
Map (num_proc=16):  24%|██▍       | 11034/46283 [00:01<00:02, 11780.43 examples/s]
Map (num_proc=16):  48%|████▊     | 22442/46283 [00:02<00:01, 12256.49 examples/s]
Map (num_proc=16):   3%|▎         | 1583/46283 [00:00<00:11, 3828.37 examples/s]
Map (num_proc=16):  51%|█████▏    | 23778/46283 [00:02<00:01, 12505.99 examples/s]
Map (num_proc=16):  26%|██▋       | 12249/46283 [00:01<00:02, 11694.71 examples/s]
Map (num_proc=16):   1%|          | 290/46283 [00:00<00:55, 827.85 examples/s]
Map (num_proc=16):   5%|▌         | 2356/46283 [00:00<00:08, 4974.99 examples/s]
Map (num_proc=16):  54%|█████▍    | 25132/46283 [00:02<00:01, 12776.99 examples/s]
Map (num_proc=16):   2%|▏         | 944/46283 [00:00<00:18, 2508.09 examples/s]
Map (num_proc=16):  29%|██▉       | 13454/46283 [00:01<00:02, 11203.84 examples/s]
Map (num_proc=16):   7%|▋         | 3331/46283 [00:00<00:06, 6372.96 examples/s]
Map (num_proc=16):  57%|█████▋    | 26447/46283 [00:02<00:01, 12764.37 examples/s]
Map (num_proc=16):   3%|▎         | 1525/46283 [00:00<00:12, 3462.94 examples/s]
Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=16):  32%|███▏      | 14610/46283 [00:01<00:02, 11175.67 examples/s]
Map (num_proc=16):  10%|▉         | 4544/46283 [00:00<00:05, 8068.36 examples/s]
Map (num_proc=16):  60%|█████▉    | 27747/46283 [00:02<00:01, 12641.81 examples/s]
Map (num_proc=16):   5%|▌         | 2479/46283 [00:00<00:08, 5192.69 examples/s]
Map (num_proc=16):  34%|███▍      | 15742/46283 [00:01<00:02, 11184.39 examples/s]
Map (num_proc=16):  12%|█▏        | 5684/46283 [00:01<00:04, 8897.35 examples/s]
Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=16):   8%|▊         | 3690/46283 [00:00<00:05, 7217.31 examples/s]
Map (num_proc=16):  63%|██████▎   | 29030/46283 [00:02<00:01, 12044.70 examples/s]
Map (num_proc=16):  36%|███▋      | 16885/46283 [00:01<00:02, 11158.03 examples/s]
Map (num_proc=16):  15%|█▌        | 7115/46283 [00:01<00:03, 10341.27 examples/s]
Map (num_proc=16):   0%|          | 29/46283 [00:00<07:49, 98.61 examples/s]
Map (num_proc=16):  10%|█         | 4826/46283 [00:00<00:04, 8406.25 examples/s]
Map (num_proc=16):  18%|█▊        | 8209/46283 [00:01<00:03, 10211.53 examples/s]
Map (num_proc=16):  39%|███▉      | 18029/46283 [00:02<00:02, 10113.92 examples/s]
Map (num_proc=16):  65%|██████▌   | 30244/46283 [00:03<00:01, 10625.92 examples/s]
Map (num_proc=16):   0%|          | 133/46283 [00:00<01:54, 401.41 examples/s]
Map (num_proc=16):  13%|█▎        | 5813/46283 [00:01<00:04, 8736.13 examples/s]
Map (num_proc=16):  20%|██        | 9271/46283 [00:01<00:03, 9935.57 examples/s] 
Map (num_proc=16):  41%|████      | 19069/46283 [00:02<00:03, 9046.55 examples/s] 
Map (num_proc=16):  15%|█▍        | 6744/46283 [00:01<00:04, 8568.84 examples/s]
Map (num_proc=16):  68%|██████▊   | 31347/46283 [00:03<00:01, 9336.29 examples/s] 
Map (num_proc=16):   1%|          | 447/46283 [00:00<00:40, 1118.11 examples/s]
Map (num_proc=16):  22%|██▏       | 10288/46283 [00:01<00:03, 9467.10 examples/s]
Map (num_proc=16):   0%|          | 20/46283 [00:00<17:18, 44.56 examples/s]
Map (num_proc=16):  17%|█▋        | 7645/46283 [00:01<00:04, 8321.29 examples/s]
Map (num_proc=16):  43%|████▎     | 20032/46283 [00:02<00:03, 8400.07 examples/s]
Map (num_proc=16):  70%|██████▉   | 32332/46283 [00:03<00:01, 8701.88 examples/s]
Map (num_proc=16):   2%|▏         | 905/46283 [00:00<00:24, 1880.85 examples/s]
Map (num_proc=16):  24%|██▍       | 11259/46283 [00:01<00:03, 9067.66 examples/s]
Map (num_proc=16):  45%|████▌     | 20915/46283 [00:02<00:03, 8407.69 examples/s]
Map (num_proc=16):  18%|█▊        | 8504/46283 [00:01<00:04, 8094.87 examples/s]
Map (num_proc=16):   0%|          | 48/46283 [00:00<08:20, 92.38 examples/s]
Map (num_proc=16):  72%|███████▏  | 33238/46283 [00:03<00:01, 8120.37 examples/s]
Map (num_proc=16):   3%|▎         | 1615/46283 [00:00<00:15, 2883.93 examples/s]
Map (num_proc=16):  26%|██▋       | 12196/46283 [00:01<00:04, 7992.16 examples/s]
Map (num_proc=16):   0%|          | 205/46283 [00:00<01:47, 427.57 examples/s]
Map (num_proc=16):  47%|████▋     | 21798/46283 [00:02<00:03, 7914.82 examples/s]
Map (num_proc=16):  20%|██        | 9342/46283 [00:01<00:04, 7515.40 examples/s]
Map (num_proc=16):   5%|▌         | 2407/46283 [00:00<00:10, 4121.85 examples/s]
Map (num_proc=16):  28%|██▊       | 13039/46283 [00:01<00:04, 7854.16 examples/s]
Map (num_proc=16):  74%|███████▎  | 34076/46283 [00:03<00:01, 6705.62 examples/s]
Map (num_proc=16):  49%|████▉     | 22614/46283 [00:02<00:03, 7765.62 examples/s]
Map (num_proc=16):  22%|██▏       | 10139/46283 [00:01<00:04, 7249.45 examples/s]
Map (num_proc=16):   1%|          | 335/46283 [00:00<01:21, 562.35 examples/s]
Map (num_proc=16):   7%|▋         | 3080/46283 [00:01<00:09, 4774.58 examples/s]
Map (num_proc=16):  30%|██▉       | 13855/46283 [00:01<00:04, 7886.75 examples/s]
Map (num_proc=16):  51%|█████     | 23435/46283 [00:02<00:02, 7822.76 examples/s]
Map (num_proc=16):  24%|██▎       | 10884/46283 [00:01<00:04, 7301.87 examples/s]
Map (num_proc=16):  75%|███████▌  | 34800/46283 [00:03<00:01, 6394.15 examples/s]
Map (num_proc=16):   8%|▊         | 3835/46283 [00:01<00:07, 5500.29 examples/s]
Map (num_proc=16):   2%|▏         | 891/46283 [00:01<00:29, 1543.36 examples/s]
Map (num_proc=16):  32%|███▏      | 14671/46283 [00:02<00:04, 7664.08 examples/s]
Map (num_proc=16):  52%|█████▏    | 24293/46283 [00:02<00:02, 7953.31 examples/s]
Map (num_proc=16):  25%|██▌       | 11691/46283 [00:01<00:04, 7494.09 examples/s]
Map (num_proc=16):  77%|███████▋  | 35497/46283 [00:03<00:01, 6211.94 examples/s]
Map (num_proc=16):  10%|▉         | 4613/46283 [00:01<00:06, 6123.74 examples/s]
Map (num_proc=16):  54%|█████▍    | 25129/46283 [00:03<00:02, 8012.77 examples/s]
Map (num_proc=16):  27%|██▋       | 12451/46283 [00:01<00:04, 7518.72 examples/s]
Map (num_proc=16):  33%|███▎      | 15470/46283 [00:02<00:04, 7293.10 examples/s]
Map (num_proc=16):   3%|▎         | 1430/46283 [00:01<00:20, 2208.03 examples/s]
Map (num_proc=16):  12%|█▏        | 5402/46283 [00:01<00:06, 6592.65 examples/s]
Map (num_proc=16):   0%|          | 0/46283 [00:00<?, ? examples/s]
Map (num_proc=16):  78%|███████▊  | 36142/46283 [00:04<00:01, 5484.87 examples/s]
Map (num_proc=16):  29%|██▊       | 13209/46283 [00:02<00:04, 7235.69 examples/s]
Map (num_proc=16):  56%|█████▌    | 25959/46283 [00:03<00:02, 7522.73 examples/s]
Map (num_proc=16):  13%|█▎        | 6105/46283 [00:01<00:06, 6608.96 examples/s]
Map (num_proc=16):  35%|███▌      | 16208/46283 [00:02<00:04, 6360.62 examples/s]
Map (num_proc=16):  79%|███████▉  | 36728/46283 [00:04<00:01, 5153.54 examples/s]
Map (num_proc=16):  15%|█▌        | 6954/46283 [00:01<00:05, 7081.76 examples/s]
Map (num_proc=16):  58%|█████▊    | 26723/46283 [00:03<00:02, 6882.03 examples/s]
Map (num_proc=16):  30%|███       | 13943/46283 [00:02<00:04, 6507.82 examples/s]
Map (num_proc=16):   4%|▍         | 1995/46283 [00:01<00:20, 2171.20 examples/s]
Map (num_proc=16):  81%|████████  | 37272/46283 [00:04<00:01, 4931.31 examples/s]
Map (num_proc=16):  17%|█▋        | 7678/46283 [00:01<00:05, 6587.23 examples/s]
Map (num_proc=16):   8%|▊         | 3616/46283 [00:01<00:08, 4866.40 examples/s]
Map (num_proc=16):  59%|█████▉    | 27431/46283 [00:03<00:02, 6321.45 examples/s]
Map (num_proc=16):  36%|███▋      | 16881/46283 [00:02<00:06, 4747.95 examples/s]
Map (num_proc=16):  32%|███▏      | 14613/46283 [00:02<00:05, 5529.09 examples/s]
Map (num_proc=16):  82%|████████▏ | 37776/46283 [00:04<00:01, 4955.90 examples/s]
Map (num_proc=16):   9%|▉         | 4303/46283 [00:01<00:08, 5146.66 examples/s]
Map (num_proc=16):  18%|█▊        | 8357/46283 [00:01<00:06, 5927.53 examples/s]
Map (num_proc=16):  61%|██████    | 28096/46283 [00:03<00:03, 5840.87 examples/s]
Map (num_proc=16):  83%|████████▎ | 38293/46283 [00:04<00:01, 4964.52 examples/s]
Map (num_proc=16):  38%|███▊      | 17436/46283 [00:02<00:06, 4402.22 examples/s]
Map (num_proc=16):  11%|█         | 4973/46283 [00:01<00:07, 5492.18 examples/s]
Map (num_proc=16):   0%|          | 33/46283 [00:00<13:04, 58.92 examples/s]
Map (num_proc=16):  33%|███▎      | 15199/46283 [00:02<00:06, 4555.09 examples/s]
Map (num_proc=16):  19%|█▉        | 8975/46283 [00:01<00:06, 5686.14 examples/s]
Map (num_proc=16):  84%|████████▍ | 38795/46283 [00:04<00:01, 4807.86 examples/s]
Map (num_proc=16):  62%|██████▏   | 28697/46283 [00:03<00:03, 5318.07 examples/s]
Map (num_proc=16):  39%|███▉      | 17943/46283 [00:02<00:06, 4367.13 examples/s]
Map (num_proc=16):  12%|█▏        | 5646/46283 [00:01<00:07, 5687.41 examples/s]
Map (num_proc=16):   0%|          | 89/46283 [00:00<04:51, 158.22 examples/s]
Map (num_proc=16):  21%|██        | 9564/46283 [00:02<00:06, 5408.51 examples/s]
Map (num_proc=16):  63%|██████▎   | 29255/46283 [00:03<00:03, 5283.12 examples/s]
Map (num_proc=16):  40%|███▉      | 18453/46283 [00:02<00:06, 4504.11 examples/s]
Map (num_proc=16):  34%|███▍      | 15705/46283 [00:02<00:07, 4111.33 examples/s]
Map (num_proc=16):  14%|█▎        | 6312/46283 [00:01<00:07, 5443.28 examples/s]
Map (num_proc=16):  85%|████████▍ | 39278/46283 [00:04<00:01, 3971.33 examples/s]
Map (num_proc=16):  22%|██▏       | 10258/46283 [00:02<00:06, 5653.85 examples/s]
Map (num_proc=16):  41%|████      | 18996/46283 [00:03<00:05, 4710.15 examples/s]
Map (num_proc=16):  35%|███▍      | 16149/46283 [00:02<00:07, 4158.64 examples/s]
Map (num_proc=16):  64%|██████▍   | 29809/46283 [00:03<00:03, 5081.18 examples/s]
Map (num_proc=16):   0%|          | 136/46283 [00:00<04:11, 183.61 examples/s]
Map (num_proc=16):  86%|████████▌ | 39713/46283 [00:05<00:01, 3952.87 examples/s]
Map (num_proc=16):  15%|█▍        | 6914/46283 [00:02<00:07, 5322.97 examples/s]
Map (num_proc=16):  24%|██▎       | 10912/46283 [00:02<00:06, 5834.10 examples/s]
Map (num_proc=16):  42%|████▏     | 19582/46283 [00:03<00:05, 4969.34 examples/s]
Map (num_proc=16):  66%|██████▌   | 30341/46283 [00:04<00:03, 4901.64 examples/s]
Map (num_proc=16):   1%|          | 332/46283 [00:00<01:25, 539.93 examples/s]
Map (num_proc=16):  36%|███▌      | 16609/46283 [00:02<00:08, 3706.99 examples/s]
Map (num_proc=16):  87%|████████▋ | 40132/46283 [00:05<00:01, 3986.34 examples/s]
Map (num_proc=16):  16%|█▌        | 7505/46283 [00:02<00:07, 5273.35 examples/s]
Map (num_proc=16):  25%|██▍       | 11526/46283 [00:02<00:06, 5336.58 examples/s]
Map (num_proc=16):  43%|████▎     | 20106/46283 [00:03<00:05, 4499.27 examples/s]
Map (num_proc=16):  67%|██████▋   | 30843/46283 [00:04<00:03, 4482.59 examples/s]
Map (num_proc=16):  37%|███▋      | 17036/46283 [00:03<00:07, 3674.42 examples/s]
Map (num_proc=16):  88%|████████▊ | 40559/46283 [00:05<00:01, 3688.09 examples/s]
Map (num_proc=16):  17%|█▋        | 8063/46283 [00:02<00:07, 4847.72 examples/s]
Map (num_proc=16):  44%|████▍     | 20583/46283 [00:03<00:05, 4395.04 examples/s]
Map (num_proc=16):  68%|██████▊   | 31320/46283 [00:04<00:03, 4395.61 examples/s]
Map (num_proc=16):  26%|██▌       | 12091/46283 [00:02<00:07, 4686.77 examples/s]
Map (num_proc=16):  38%|███▊      | 17423/46283 [00:03<00:08, 3539.71 examples/s]
Map (num_proc=16):  89%|████████▊ | 40996/46283 [00:05<00:01, 3832.77 examples/s]
Map (num_proc=16):  19%|█▊        | 8577/46283 [00:02<00:07, 4872.26 examples/s]
Map (num_proc=16):  46%|████▌     | 21115/46283 [00:03<00:05, 4558.09 examples/s]
Map (num_proc=16):  27%|██▋       | 12597/46283 [00:02<00:07, 4747.67 examples/s]
Map (num_proc=16):  38%|███▊      | 17792/46283 [00:03<00:08, 3559.91 examples/s]
Map (num_proc=16):  89%|████████▉ | 41405/46283 [00:05<00:01, 3881.69 examples/s]
Map (num_proc=16):   1%|          | 548/46283 [00:01<01:24, 543.45 examples/s]
Map (num_proc=16):  20%|█▉        | 9110/46283 [00:02<00:07, 4956.56 examples/s]
Map (num_proc=16):  69%|██████▊   | 31770/46283 [00:04<00:03, 3832.25 examples/s]
Map (num_proc=16):  29%|██▊       | 13219/46283 [00:02<00:06, 5103.80 examples/s]
Map (num_proc=16):  39%|███▉      | 18265/46283 [00:03<00:07, 3815.50 examples/s]
Map (num_proc=16):  47%|████▋     | 21602/46283 [00:03<00:05, 4128.61 examples/s]
Map (num_proc=16):  70%|██████▉   | 32170/46283 [00:04<00:03, 3840.99 examples/s]
Map (num_proc=16):  21%|██        | 9626/46283 [00:02<00:07, 4722.86 examples/s]
Map (num_proc=16):  90%|█████████ | 41801/46283 [00:05<00:01, 3378.77 examples/s]
Map (num_proc=16):   3%|▎         | 1247/46283 [00:01<00:31, 1432.16 examples/s]
Map (num_proc=16):  30%|██▉       | 13776/46283 [00:02<00:06, 5058.37 examples/s]
Map (num_proc=16):  41%|████      | 18833/46283 [00:03<00:06, 4273.19 examples/s]
Map (num_proc=16):  70%|███████   | 32583/46283 [00:04<00:03, 3690.59 examples/s]
Map (num_proc=16):  48%|████▊     | 22034/46283 [00:03<00:06, 3492.14 examples/s]
Map (num_proc=16):  22%|██▏       | 10115/46283 [00:02<00:08, 4312.97 examples/s]
Map (num_proc=16):  31%|███       | 14293/46283 [00:03<00:06, 4993.45 examples/s]
Map (num_proc=16):  91%|█████████ | 42168/46283 [00:05<00:01, 2965.90 examples/s]
Map (num_proc=16):  42%|████▏     | 19278/46283 [00:03<00:06, 3930.44 examples/s]
Map (num_proc=16):  71%|███████▏  | 32980/46283 [00:04<00:03, 3562.21 examples/s]
Map (num_proc=16):  48%|████▊     | 22411/46283 [00:03<00:06, 3449.97 examples/s]
Map (num_proc=16):  23%|██▎       | 10839/46283 [00:02<00:07, 4968.51 examples/s]
Map (num_proc=16):   3%|▎         | 1595/46283 [00:01<00:29, 1491.90 examples/s]
Map (num_proc=16):  43%|████▎     | 19737/46283 [00:03<00:06, 4073.00 examples/s]
Map (num_proc=16):  32%|███▏      | 14810/46283 [00:03<00:07, 4251.48 examples/s]
Map (num_proc=16):  72%|███████▏  | 33410/46283 [00:04<00:03, 3739.01 examples/s]
Map (num_proc=16):  50%|████▉     | 22911/46283 [00:04<00:06, 3818.38 examples/s]
Map (num_proc=16):  92%|█████████▏| 42484/46283 [00:05<00:01, 2538.28 examples/s]
Map (num_proc=16):  25%|██▍       | 11359/46283 [00:03<00:07, 4754.65 examples/s]
Map (num_proc=16):  44%|████▍     | 20261/46283 [00:03<00:05, 4376.74 examples/s]
Map (num_proc=16):  33%|███▎      | 15282/46283 [00:03<00:07, 4120.41 examples/s]
Map (num_proc=16):  50%|█████     | 23334/46283 [00:04<00:05, 3899.24 examples/s]
Map (num_proc=16):  73%|███████▎  | 33814/46283 [00:05<00:03, 3738.34 examples/s]
Map (num_proc=16):  92%|█████████▏| 42768/46283 [00:06<00:01, 2559.22 examples/s]
Map (num_proc=16):  26%|██▌       | 11851/46283 [00:03<00:07, 4626.48 examples/s]
Map (num_proc=16):  45%|████▍     | 20805/46283 [00:03<00:05, 4671.03 examples/s]
Map (num_proc=16):   5%|▍         | 2204/46283 [00:01<00:23, 1838.11 examples/s]
Map (num_proc=16):  74%|███████▍  | 34254/46283 [00:05<00:03, 3896.08 examples/s]
Map (num_proc=16):  51%|█████▏    | 23774/46283 [00:04<00:05, 4025.33 examples/s]
Map (num_proc=16):  34%|███▍      | 15719/46283 [00:03<00:07, 4127.89 examples/s]
Map (num_proc=16):  46%|████▌     | 21344/46283 [00:04<00:05, 4870.25 examples/s]
Map (num_proc=16):   8%|▊         | 3610/46283 [00:02<00:11, 3823.27 examples/s]
Map (num_proc=16):  27%|██▋       | 12322/46283 [00:03<00:07, 4355.09 examples/s]
Map (num_proc=16):  93%|█████████▎| 43063/46283 [00:06<00:01, 2404.42 examples/s]
Map (num_proc=16):  75%|███████▌  | 34872/46283 [00:05<00:02, 4511.77 examples/s]
Map (num_proc=16):  53%|█████▎    | 24406/46283 [00:04<00:04, 4623.73 examples/s]
Map (num_proc=16):  35%|███▌      | 16301/46283 [00:03<00:06, 4539.55 examples/s]
Map (num_proc=16):  94%|█████████▎| 43370/46283 [00:06<00:01, 2548.39 examples/s]
Map (num_proc=16):  47%|████▋     | 21871/46283 [00:04<00:05, 4645.95 examples/s]
Map (num_proc=16):  28%|██▊       | 12784/46283 [00:03<00:08, 4172.99 examples/s]
Map (num_proc=16):  54%|█████▍    | 24932/46283 [00:04<00:04, 4800.79 examples/s]
Map (num_proc=16):  76%|███████▋  | 35347/46283 [00:05<00:02, 4440.90 examples/s]
Map (num_proc=16):  36%|███▋      | 16793/46283 [00:03<00:07, 4000.05 examples/s]
Map (num_proc=16):  94%|█████████▍| 43716/46283 [00:06<00:00, 2736.66 examples/s]
Map (num_proc=16):  49%|████▊     | 22489/46283 [00:04<00:04, 4952.75 examples/s]
Map (num_proc=16):  55%|█████▌    | 25469/46283 [00:04<00:04, 4921.87 examples/s]
Map (num_proc=16):  29%|██▊       | 13207/46283 [00:03<00:08, 3875.94 examples/s]
Map (num_proc=16):  77%|███████▋  | 35824/46283 [00:05<00:02, 4137.79 examples/s]
Map (num_proc=16):   9%|▉         | 4195/46283 [00:02<00:13, 3049.51 examples/s]
Map (num_proc=16):  95%|█████████▌| 44059/46283 [00:06<00:00, 2854.72 examples/s]
Map (num_proc=16):  50%|████▉     | 23029/46283 [00:04<00:04, 4976.32 examples/s]
Map (num_proc=16):  56%|█████▌    | 26023/46283 [00:04<00:04, 5039.72 examples/s]
Map (num_proc=16):  37%|███▋      | 17213/46283 [00:03<00:08, 3579.72 examples/s]
Map (num_proc=16):  30%|██▉       | 13658/46283 [00:03<00:08, 4006.42 examples/s]
Map (num_proc=16):  11%|█▏        | 5239/46283 [00:02<00:09, 4206.05 examples/s]
Map (num_proc=16):  96%|█████████▌| 44360/46283 [00:06<00:00, 2842.74 examples/s]
Map (num_proc=16):  51%|█████     | 23566/46283 [00:04<00:04, 5071.11 examples/s]
Map (num_proc=16):  78%|███████▊  | 36247/46283 [00:05<00:02, 3695.59 examples/s]
Map (num_proc=16):  57%|█████▋    | 26553/46283 [00:04<00:03, 5076.32 examples/s]
Map (num_proc=16):  38%|███▊      | 17593/46283 [00:03<00:08, 3304.94 examples/s]
Map (num_proc=16):  30%|███       | 14069/46283 [00:03<00:09, 3513.85 examples/s]
Map (num_proc=16):  13%|█▎        | 5864/46283 [00:02<00:09, 4395.71 examples/s]
Map (num_proc=16):  52%|█████▏    | 24124/46283 [00:04<00:04, 5165.24 examples/s]
Map (num_proc=16):  79%|███████▉  | 36636/46283 [00:05<00:02, 3705.30 examples/s]
Map (num_proc=16):  96%|█████████▋| 44651/46283 [00:06<00:00, 2756.95 examples/s]
Map (num_proc=16):  59%|█████▊    | 27181/46283 [00:04<00:03, 5409.77 examples/s]
Map (num_proc=16):  31%|███       | 14455/46283 [00:03<00:08, 3589.87 examples/s]
Map (num_proc=16):  39%|███▉      | 17944/46283 [00:04<00:08, 3166.93 examples/s]
Map (num_proc=16):  14%|█▍        | 6457/46283 [00:02<00:08, 4591.62 examples/s]
Map (num_proc=16):  53%|█████▎    | 24748/46283 [00:04<00:04, 5344.73 examples/s]
Map (num_proc=16):  80%|████████  | 37054/46283 [00:05<00:02, 3780.08 examples/s]
Map (num_proc=16):  60%|█████▉    | 27736/46283 [00:05<00:03, 5376.19 examples/s]
Map (num_proc=16):  97%|█████████▋| 44931/46283 [00:06<00:00, 2357.46 examples/s]
Map (num_proc=16):  32%|███▏      | 14931/46283 [00:04<00:08, 3875.60 examples/s]
Map (num_proc=16):  15%|█▌        | 7042/46283 [00:02<00:08, 4765.43 examples/s]
Map (num_proc=16):  55%|█████▍    | 25291/46283 [00:04<00:04, 5028.94 examples/s]
Map (num_proc=16):  61%|██████▏   | 28432/46283 [00:05<00:03, 5745.39 examples/s]
Map (num_proc=16):  40%|███▉      | 18286/46283 [00:04<00:09, 2922.24 examples/s]
Map (num_proc=16):  81%|████████  | 37439/46283 [00:06<00:02, 3414.31 examples/s]
Map (num_proc=16):  33%|███▎      | 15404/46283 [00:04<00:07, 4067.65 examples/s]
Map (num_proc=16):  56%|█████▌    | 25819/46283 [00:04<00:04, 4960.40 examples/s]
Map (num_proc=16):  16%|█▋        | 7599/46283 [00:02<00:08, 4509.40 examples/s]
Map (num_proc=16):  40%|████      | 18590/46283 [00:04<00:10, 2743.31 examples/s]
Map (num_proc=16):  63%|██████▎   | 29021/46283 [00:05<00:03, 5253.15 examples/s]
Map (num_proc=16):  82%|████████▏ | 37795/46283 [00:06<00:02, 3259.76 examples/s]
Map (num_proc=16):  34%|███▍      | 15924/46283 [00:04<00:07, 3982.67 examples/s]
Map (num_proc=16):  98%|█████████▊| 45187/46283 [00:07<00:00, 1730.67 examples/s]
Map (num_proc=16):  57%|█████▋    | 26320/46283 [00:05<00:04, 4905.84 examples/s]
Map (num_proc=16):  41%|████      | 18889/46283 [00:04<00:09, 2802.04 examples/s]
Map (num_proc=16):  83%|████████▎ | 38265/46283 [00:06<00:02, 3605.27 examples/s]
Map (num_proc=16):  64%|██████▍   | 29565/46283 [00:05<00:03, 5176.69 examples/s]
Map (num_proc=16):  18%|█▊        | 8120/46283 [00:03<00:09, 4143.94 examples/s]
Map (num_proc=16):  36%|███▌      | 16598/46283 [00:04<00:06, 4704.93 examples/s]
Map (num_proc=16):  98%|█████████▊| 45411/46283 [00:07<00:00, 1802.75 examples/s]
Map (num_proc=16):  58%|█████▊    | 26813/46283 [00:05<00:04, 4782.40 examples/s]
Map (num_proc=16):  41%|████▏     | 19206/46283 [00:04<00:09, 2880.76 examples/s]
Map (num_proc=16):  84%|████████▍ | 38786/46283 [00:06<00:01, 3954.89 examples/s]
Map (num_proc=16):  65%|██████▌   | 30090/46283 [00:05<00:03, 4832.07 examples/s]
Map (num_proc=16):  19%|█▉        | 8795/46283 [00:03<00:08, 4677.63 examples/s]
Map (num_proc=16):  37%|███▋      | 17089/46283 [00:04<00:06, 4748.66 examples/s]
Map (num_proc=16):  99%|█████████▊| 45620/46283 [00:07<00:00, 1864.03 examples/s]
Map (num_proc=16):  59%|█████▉    | 27294/46283 [00:05<00:04, 4666.54 examples/s]
Map (num_proc=16):  43%|████▎     | 19703/46283 [00:04<00:07, 3425.81 examples/s]
Map (num_proc=16):  85%|████████▍ | 39250/46283 [00:06<00:01, 4116.14 examples/s]
Map (num_proc=16):  20%|██        | 9359/46283 [00:03<00:07, 4833.75 examples/s]
Map (num_proc=16):  66%|██████▌   | 30604/46283 [00:05<00:03, 4551.15 examples/s]
Map (num_proc=16):  99%|█████████▉| 45845/46283 [00:07<00:00, 1939.76 examples/s]
Map (num_proc=16):  38%|███▊      | 17583/46283 [00:04<00:06, 4522.72 examples/s]
Map (num_proc=16):  44%|████▎     | 20201/46283 [00:04<00:06, 3832.39 examples/s]
Map (num_proc=16):  86%|████████▌ | 39695/46283 [00:06<00:01, 4148.90 examples/s]
Map (num_proc=16):  60%|██████    | 27785/46283 [00:05<00:04, 4397.68 examples/s]
Map (num_proc=16):  21%|██▏       | 9879/46283 [00:03<00:07, 4873.37 examples/s]
Map (num_proc=16):  39%|███▉      | 18077/46283 [00:04<00:06, 4316.97 examples/s]
Map (num_proc=16):  45%|████▍     | 20623/46283 [00:04<00:06, 3860.18 examples/s]
Map (num_proc=16):  67%|██████▋   | 31073/46283 [00:05<00:03, 4096.52 examples/s]
Map (num_proc=16):  61%|██████    | 28229/46283 [00:05<00:04, 4287.99 examples/s]
Map (num_proc=16):  87%|████████▋ | 40135/46283 [00:06<00:01, 3847.27 examples/s]
Map (num_proc=16):  22%|██▏       | 10402/46283 [00:03<00:07, 4714.82 examples/s]
Map (num_proc=16):  45%|████▌     | 21020/46283 [00:04<00:06, 3887.08 examples/s]
Map (num_proc=16):  62%|██████▏   | 28676/46283 [00:05<00:04, 4329.09 examples/s]
Map (num_proc=16):  68%|██████▊   | 31497/46283 [00:05<00:03, 3769.69 examples/s]
Map (num_proc=16):  40%|████      | 18526/46283 [00:04<00:07, 3882.54 examples/s]
Map (num_proc=16):  88%|████████▊ | 40650/46283 [00:06<00:01, 4168.45 examples/s]
Map (num_proc=16): 100%|█████████▉| 46074/46283 [00:07<00:00, 1338.12 examples/s]
Map (num_proc=16):  24%|██▎       | 10945/46283 [00:03<00:07, 4851.94 examples/s]
Map (num_proc=16):  47%|████▋     | 21538/46283 [00:05<00:05, 4237.02 examples/s]
Map (num_proc=16):  63%|██████▎   | 29127/46283 [00:05<00:03, 4369.81 examples/s]
Map (num_proc=16):  69%|██████▉   | 32022/46283 [00:06<00:03, 4122.61 examples/s]
Map (num_proc=16):  89%|████████▉ | 41084/46283 [00:06<00:01, 4205.87 examples/s]
Map (num_proc=16):  41%|████      | 18929/46283 [00:04<00:07, 3724.48 examples/s]
Map (num_proc=16):  25%|██▍       | 11468/46283 [00:03<00:07, 4807.82 examples/s]
Map (num_proc=16):  48%|████▊     | 22144/46283 [00:05<00:05, 4728.77 examples/s]
Map (num_proc=16):  64%|██████▍   | 29582/46283 [00:05<00:03, 4379.27 examples/s]
Map (num_proc=16):  70%|███████   | 32480/46283 [00:06<00:03, 4210.49 examples/s]
Map (num_proc=16):  42%|████▏     | 19310/46283 [00:05<00:07, 3707.77 examples/s]
Map (num_proc=16):  90%|████████▉ | 41514/46283 [00:07<00:01, 3882.58 examples/s]
Map (num_proc=16):  26%|██▌       | 12023/46283 [00:03<00:06, 4987.64 examples/s]
Map (num_proc=16):  49%|████▉     | 22679/46283 [00:05<00:04, 4772.93 examples/s]
Map (num_proc=16):  65%|██████▍   | 30026/46283 [00:05<00:03, 4391.19 examples/s]
Map (num_proc=16):  71%|███████▏  | 32984/46283 [00:06<00:03, 4400.56 examples/s]
Map (num_proc=16):  43%|████▎     | 19735/46283 [00:05<00:07, 3761.21 examples/s]
Map (num_proc=16):  27%|██▋       | 12673/46283 [00:04<00:06, 5409.97 examples/s]
Map (num_proc=16):  91%|█████████ | 41942/46283 [00:07<00:01, 3906.07 examples/s]
Map (num_proc=16):  50%|█████     | 23205/46283 [00:05<00:04, 4880.07 examples/s]
Map (num_proc=16):  66%|██████▌   | 30528/46283 [00:06<00:03, 4567.45 examples/s]
Map (num_proc=16):  72%|███████▏  | 33437/46283 [00:06<00:03, 4188.01 examples/s]
Map (num_proc=16):  44%|████▎     | 20208/46283 [00:05<00:06, 4022.70 examples/s]
Map (num_proc=16):  29%|██▊       | 13248/46283 [00:04<00:06, 5479.99 examples/s]
Map (num_proc=16):  92%|█████████▏| 42411/46283 [00:07<00:00, 4109.00 examples/s]
Map (num_proc=16):  51%|█████▏    | 23787/46283 [00:05<00:04, 5143.16 examples/s]
Map (num_proc=16):  67%|██████▋   | 31192/46283 [00:06<00:02, 5155.96 examples/s]
Map (num_proc=16):  73%|███████▎  | 33865/46283 [00:06<00:03, 4079.62 examples/s]
Map (num_proc=16):  45%|████▍     | 20634/46283 [00:05<00:06, 4051.55 examples/s]
Map (num_proc=16):  93%|█████████▎| 42880/46283 [00:07<00:00, 4259.93 examples/s]
Map (num_proc=16):  53%|█████▎    | 24433/46283 [00:05<00:03, 5498.47 examples/s]
Map (num_proc=16):  69%|██████▊   | 31810/46283 [00:06<00:02, 5391.86 examples/s]
Map (num_proc=16):  30%|██▉       | 13830/46283 [00:04<00:06, 4822.13 examples/s]
Map (num_proc=16):  46%|████▌     | 21205/46283 [00:05<00:05, 4500.29 examples/s]
Map (num_proc=16):  94%|█████████▎| 43341/46283 [00:07<00:00, 4306.49 examples/s]
Map (num_proc=16):  54%|█████▍    | 25015/46283 [00:05<00:03, 5536.94 examples/s]
Map (num_proc=16):  74%|███████▍  | 34282/46283 [00:06<00:03, 3652.91 examples/s]
Map (num_proc=16):  70%|██████▉   | 32352/46283 [00:06<00:02, 5343.65 examples/s]
Map (num_proc=16):  31%|███       | 14337/46283 [00:04<00:06, 4706.55 examples/s]
Map (num_proc=16):  47%|████▋     | 21728/46283 [00:05<00:05, 4630.24 examples/s]
Map (num_proc=16):  95%|█████████▍| 43783/46283 [00:07<00:00, 4176.00 examples/s]
Map (num_proc=16): 100%|█████████▉| 46247/46283 [00:08<00:00, 615.65 examples/s] 
Map (num_proc=16):  75%|███████▍  | 34691/46283 [00:06<00:03, 3602.79 examples/s]
Map (num_proc=16):  55%|█████▌    | 25574/46283 [00:05<00:04, 5134.21 examples/s]
Map (num_proc=16):  32%|███▏      | 14824/46283 [00:04<00:06, 4678.40 examples/s]
Map (num_proc=16):  71%|███████   | 32916/46283 [00:06<00:02, 4836.56 examples/s]
Map (num_proc=16):  48%|████▊     | 22204/46283 [00:05<00:05, 4648.53 examples/s]
Map (num_proc=16):  76%|███████▌  | 35064/46283 [00:06<00:03, 3424.36 examples/s]
Map (num_proc=16):  33%|███▎      | 15321/46283 [00:04<00:06, 4639.40 examples/s]
Map (num_proc=16):  56%|█████▋    | 26114/46283 [00:05<00:04, 4818.19 examples/s]
Map (num_proc=16):  96%|█████████▌| 44235/46283 [00:07<00:00, 3541.92 examples/s]
Map (num_proc=16):  49%|████▉     | 22682/46283 [00:05<00:05, 4684.40 examples/s]
Map (num_proc=16):  72%|███████▏  | 33412/46283 [00:06<00:02, 4484.02 examples/s]
Map (num_proc=16):  77%|███████▋  | 35575/46283 [00:06<00:02, 3837.03 examples/s]
Map (num_proc=16):  57%|█████▋    | 26612/46283 [00:06<00:04, 4835.11 examples/s]
Map (num_proc=16):  34%|███▍      | 15808/46283 [00:04<00:06, 4465.12 examples/s]
Map (num_proc=16):  50%|█████     | 23175/46283 [00:05<00:04, 4754.93 examples/s]
Map (num_proc=16):  96%|█████████▋| 44626/46283 [00:07<00:00, 3440.13 examples/s]
Map (num_proc=16):  73%|███████▎  | 33886/46283 [00:06<00:02, 4278.22 examples/s]
Map (num_proc=16):  78%|███████▊  | 36098/46283 [00:07<00:02, 4196.39 examples/s]
Map (num_proc=16):  35%|███▌      | 16318/46283 [00:04<00:06, 4637.53 examples/s]
Map (num_proc=16):  51%|█████     | 23658/46283 [00:06<00:04, 4545.22 examples/s]
Map (num_proc=16):  59%|█████▊    | 27102/46283 [00:06<00:04, 4518.74 examples/s]
Map (num_proc=16):  74%|███████▍  | 34408/46283 [00:06<00:02, 4520.93 examples/s]
Map (num_proc=16):  79%|███████▉  | 36647/46283 [00:07<00:02, 4540.59 examples/s]
Map (num_proc=16):  97%|█████████▋| 44985/46283 [00:08<00:00, 2887.87 examples/s]
Map (num_proc=16):  36%|███▋      | 16792/46283 [00:04<00:06, 4597.44 examples/s]
Map (num_proc=16):  60%|█████▉    | 27597/46283 [00:06<00:04, 4629.57 examples/s]
Map (num_proc=16):  52%|█████▏    | 24117/46283 [00:06<00:05, 4355.05 examples/s]
Map (num_proc=16):  75%|███████▌  | 34877/46283 [00:06<00:02, 4538.18 examples/s]
Map (num_proc=16):  80%|████████  | 37189/46283 [00:07<00:01, 4741.95 examples/s]
Map (num_proc=16):  98%|█████████▊| 45305/46283 [00:08<00:00, 2941.42 examples/s]
Map (num_proc=16):  37%|███▋      | 17273/46283 [00:05<00:06, 4639.07 examples/s]
Map (num_proc=16):  53%|█████▎    | 24635/46283 [00:06<00:04, 4564.14 examples/s]
Map (num_proc=16):  61%|██████    | 28099/46283 [00:06<00:04, 4459.96 examples/s]
Map (num_proc=16):  77%|███████▋  | 35422/46283 [00:07<00:02, 4767.62 examples/s]
Map (num_proc=16):  82%|████████▏ | 37830/46283 [00:07<00:01, 5135.41 examples/s]
Map (num_proc=16):  39%|███▊      | 17894/46283 [00:05<00:05, 5088.28 examples/s]
Map (num_proc=16):  99%|█████████▊| 45616/46283 [00:08<00:00, 2781.22 examples/s]
Map (num_proc=16):  55%|█████▍    | 25282/46283 [00:06<00:04, 5094.06 examples/s]
Map (num_proc=16):  62%|██████▏   | 28578/46283 [00:06<00:03, 4533.92 examples/s]
Map (num_proc=16):  78%|███████▊  | 35918/46283 [00:07<00:02, 4794.57 examples/s]
Map (num_proc=16):  83%|████████▎ | 38394/46283 [00:07<00:01, 5278.91 examples/s]
Map (num_proc=16):  40%|████      | 18600/46283 [00:05<00:04, 5633.62 examples/s]
Map (num_proc=16):  56%|█████▌    | 25920/46283 [00:06<00:03, 5455.87 examples/s]
Map (num_proc=16):  63%|██████▎   | 29132/46283 [00:06<00:03, 4807.85 examples/s]
Map (num_proc=16):  79%|███████▉  | 36459/46283 [00:07<00:01, 4963.87 examples/s]
Map (num_proc=16):  84%|████████▍ | 38986/46283 [00:07<00:01, 5397.83 examples/s]
Map (num_proc=16):  99%|█████████▉| 45914/46283 [00:08<00:00, 2387.50 examples/s]
Map (num_proc=16):  41%|████▏     | 19197/46283 [00:05<00:04, 5699.66 examples/s]
Map (num_proc=16):  57%|█████▋    | 26599/46283 [00:06<00:03, 5838.57 examples/s]
Map (num_proc=16):  64%|██████▍   | 29685/46283 [00:06<00:03, 4997.15 examples/s]
Map (num_proc=16):  80%|████████  | 37056/46283 [00:07<00:01, 5241.93 examples/s]
Map (num_proc=16):  86%|████████▌ | 39633/46283 [00:07<00:01, 5692.71 examples/s]
Map (num_proc=16):  43%|████▎     | 19791/46283 [00:05<00:04, 5754.34 examples/s]
                                                                                

Map (num_proc=16):  59%|█████▉    | 27254/46283 [00:06<00:03, 6040.30 examples/s]
Map (num_proc=16):  66%|██████▌   | 30504/46283 [00:06<00:02, 5904.58 examples/s]
Map (num_proc=16):  81%|████████▏ | 37680/46283 [00:07<00:01, 5518.09 examples/s]
Map (num_proc=16):  87%|████████▋ | 40288/46283 [00:07<00:01, 5864.27 examples/s]
Map (num_proc=16):  44%|████▍     | 20373/46283 [00:05<00:04, 5657.61 examples/s]
Map (num_proc=16):  60%|██████    | 27958/46283 [00:06<00:02, 6314.10 examples/s]
Map (num_proc=16):  67%|██████▋   | 31185/46283 [00:06<00:02, 6113.47 examples/s]
Map (num_proc=16):  83%|████████▎ | 38301/46283 [00:07<00:01, 5714.24 examples/s]
Map (num_proc=16): 100%|█████████▉| 46194/46283 [00:08<00:00, 1794.82 examples/s]
Map (num_proc=16):  88%|████████▊ | 40885/46283 [00:07<00:00, 5511.51 examples/s]
Map (num_proc=16):  45%|████▌     | 20947/46283 [00:05<00:04, 5566.15 examples/s]
Map (num_proc=16):  62%|██████▏   | 28595/46283 [00:06<00:02, 6297.24 examples/s]
Map (num_proc=16):  69%|██████▉   | 31827/46283 [00:07<00:02, 5697.40 examples/s]
Map (num_proc=16):  84%|████████▍ | 38878/46283 [00:07<00:01, 5235.93 examples/s]
Map (num_proc=16):  46%|████▋     | 21520/46283 [00:05<00:04, 5391.23 examples/s]
Map (num_proc=16):  63%|██████▎   | 29269/46283 [00:06<00:02, 6344.61 examples/s]
Map (num_proc=16):  90%|████████▉ | 41473/46283 [00:07<00:00, 5282.61 examples/s]
Map (num_proc=16):  70%|███████   | 32443/46283 [00:07<00:02, 5762.84 examples/s]
Map (num_proc=16):  85%|████████▌ | 39414/46283 [00:07<00:01, 5216.36 examples/s]
Map (num_proc=16):  48%|████▊     | 22108/46283 [00:05<00:04, 5474.56 examples/s]
Map (num_proc=16):  91%|█████████ | 42071/46283 [00:08<00:00, 5472.41 examples/s]
Map (num_proc=16):  65%|██████▍   | 29910/46283 [00:07<00:02, 6158.54 examples/s]
Map (num_proc=16):  72%|███████▏  | 33122/46283 [00:07<00:02, 6010.68 examples/s]
Map (num_proc=16):  86%|████████▋ | 39985/46283 [00:07<00:01, 5240.51 examples/s]
Map (num_proc=16):  49%|████▉     | 22796/46283 [00:05<00:04, 5813.21 examples/s]
Map (num_proc=16):  66%|██████▌   | 30590/46283 [00:07<00:02, 6279.24 examples/s]
Map (num_proc=16):  92%|█████████▏| 42629/46283 [00:08<00:00, 5194.35 examples/s]
Map (num_proc=16):  73%|███████▎  | 33731/46283 [00:07<00:02, 5983.94 examples/s]
Map (num_proc=16):  88%|████████▊ | 40572/46283 [00:08<00:01, 5330.87 examples/s]
                                                                                 

Map (num_proc=16):  51%|█████     | 23392/46283 [00:06<00:03, 5805.90 examples/s]
Map (num_proc=16):  68%|██████▊   | 31248/46283 [00:07<00:02, 6247.97 examples/s]
Map (num_proc=16):  93%|█████████▎| 43162/46283 [00:08<00:00, 4739.71 examples/s]
Map (num_proc=16):  74%|███████▍  | 34353/46283 [00:07<00:02, 5746.54 examples/s]
Map (num_proc=16):  89%|████████▉ | 41118/46283 [00:08<00:01, 5083.14 examples/s]
Map (num_proc=16):  52%|█████▏    | 24226/46283 [00:06<00:03, 6535.32 examples/s]
Map (num_proc=16):  69%|██████▉   | 31999/46283 [00:07<00:02, 6590.68 examples/s]
Map (num_proc=16):  94%|█████████▍| 43646/46283 [00:08<00:00, 4754.77 examples/s]
Map (num_proc=16):  76%|███████▌  | 35129/46283 [00:07<00:01, 6287.82 examples/s]
Map (num_proc=16):  90%|████████▉ | 41641/46283 [00:08<00:00, 4852.45 examples/s]
Map (num_proc=16):  54%|█████▍    | 24972/46283 [00:06<00:03, 6788.60 examples/s]
Map (num_proc=16):  71%|███████   | 32954/46283 [00:07<00:01, 7389.99 examples/s]
Map (num_proc=16):  95%|█████████▌| 44166/46283 [00:08<00:00, 4853.92 examples/s]
Map (num_proc=16):  78%|███████▊  | 35974/46283 [00:07<00:01, 6884.65 examples/s]
Map (num_proc=16):  91%|█████████ | 42155/46283 [00:08<00:00, 4912.22 examples/s]
Map (num_proc=16):  56%|█████▌    | 25816/46283 [00:06<00:02, 7267.89 examples/s]
Map (num_proc=16):  73%|███████▎  | 33825/46283 [00:07<00:01, 7601.76 examples/s]
Map (num_proc=16):  80%|███████▉  | 36832/46283 [00:07<00:01, 7352.88 examples/s]
Map (num_proc=16):  97%|█████████▋| 44668/46283 [00:08<00:00, 4828.90 examples/s]
Map (num_proc=16):  92%|█████████▏| 42725/46283 [00:08<00:00, 5075.94 examples/s]
Map (num_proc=16):  58%|█████▊    | 26806/46283 [00:06<00:02, 8019.33 examples/s]
Map (num_proc=16):  75%|███████▍  | 34611/46283 [00:07<00:01, 7533.44 examples/s]
Map (num_proc=16):  81%|████████▏ | 37663/46283 [00:07<00:01, 7591.27 examples/s]
Map (num_proc=16):  98%|█████████▊| 45165/46283 [00:08<00:00, 4570.10 examples/s]
Map (num_proc=16):  94%|█████████▎| 43307/46283 [00:08<00:00, 5258.25 examples/s]
Map (num_proc=16):  60%|█████▉    | 27646/46283 [00:06<00:02, 8036.47 examples/s]
Map (num_proc=16):  77%|███████▋  | 35499/46283 [00:07<00:01, 7876.42 examples/s]
Map (num_proc=16):  83%|████████▎ | 38450/46283 [00:07<00:01, 7641.01 examples/s]
Map (num_proc=16):  99%|█████████▊| 45650/46283 [00:08<00:00, 4429.24 examples/s]
Map (num_proc=16):  95%|█████████▍| 43844/46283 [00:08<00:00, 5223.12 examples/s]
Map (num_proc=16):  62%|██████▏   | 28474/46283 [00:06<00:02, 8020.81 examples/s]
Map (num_proc=16):  79%|███████▊  | 36409/46283 [00:07<00:01, 8125.20 examples/s]
Map (num_proc=16):  85%|████████▍ | 39224/46283 [00:08<00:00, 7302.97 examples/s]
Map (num_proc=16):  96%|█████████▌| 44375/46283 [00:08<00:00, 5192.55 examples/s]
Map (num_proc=16):  63%|██████▎   | 29302/46283 [00:06<00:02, 8010.89 examples/s]
Map (num_proc=16):  80%|████████  | 37250/46283 [00:07<00:01, 8160.77 examples/s]
Map (num_proc=16):  86%|████████▋ | 39967/46283 [00:08<00:00, 7067.55 examples/s]
Map (num_proc=16):  66%|██████▌   | 30394/46283 [00:06<00:01, 8822.03 examples/s]
Map (num_proc=16):  83%|████████▎ | 38259/46283 [00:08<00:00, 8689.38 examples/s]
Map (num_proc=16):  97%|█████████▋| 44911/46283 [00:08<00:00, 4644.93 examples/s]
Map (num_proc=16): 100%|█████████▉| 46115/46283 [00:09<00:00, 3162.25 examples/s]
Map (num_proc=16):  88%|████████▊ | 40733/46283 [00:08<00:00, 7232.20 examples/s]
Map (num_proc=16):  68%|██████▊   | 31517/46283 [00:06<00:01, 9506.33 examples/s]
Map (num_proc=16):  85%|████████▍ | 39340/46283 [00:08<00:00, 9279.47 examples/s]
Map (num_proc=16):  90%|████████▉ | 41563/46283 [00:08<00:00, 7481.10 examples/s]
Map (num_proc=16):  98%|█████████▊| 45420/46283 [00:09<00:00, 4109.66 examples/s]
Map (num_proc=16):  70%|███████   | 32567/46283 [00:07<00:01, 9702.04 examples/s]
Map (num_proc=16):  87%|████████▋ | 40389/46283 [00:08<00:00, 9616.76 examples/s]
Map (num_proc=16):  92%|█████████▏| 42368/46283 [00:08<00:00, 7612.66 examples/s]
Map (num_proc=16):  73%|███████▎  | 33784/46283 [00:07<00:01, 10381.81 examples/s]
Map (num_proc=16):  90%|████████▉ | 41429/46283 [00:08<00:00, 9780.08 examples/s]
                                                                                 

Map (num_proc=16):  93%|█████████▎| 43155/46283 [00:08<00:00, 7557.79 examples/s]
Map (num_proc=16):  76%|███████▌  | 35111/46283 [00:07<00:01, 11170.74 examples/s]
Map (num_proc=16):  92%|█████████▏| 42558/46283 [00:08<00:00, 10095.95 examples/s]
Map (num_proc=16):  99%|█████████▉| 45873/46283 [00:09<00:00, 3051.62 examples/s]
Map (num_proc=16):  95%|█████████▍| 43930/46283 [00:08<00:00, 7547.51 examples/s]
Map (num_proc=16):  79%|███████▊  | 36434/46283 [00:07<00:00, 11690.78 examples/s]
Map (num_proc=16):  94%|█████████▍| 43592/46283 [00:08<00:00, 9615.37 examples/s] 
Map (num_proc=16):  81%|████████▏ | 37638/46283 [00:07<00:00, 11720.01 examples/s]
Map (num_proc=16):  97%|█████████▋| 44710/46283 [00:08<00:00, 6614.71 examples/s]
Map (num_proc=16):  96%|█████████▋| 44600/46283 [00:08<00:00, 9339.45 examples/s]
Map (num_proc=16): 100%|█████████▉| 46264/46283 [00:09<00:00, 2490.11 examples/s]
Map (num_proc=16):  84%|████████▍ | 38920/46283 [00:07<00:00, 11888.21 examples/s]
Map (num_proc=16):  87%|████████▋ | 40148/46283 [00:07<00:00, 11594.37 examples/s]
Map (num_proc=16):  98%|█████████▊| 45564/46283 [00:08<00:00, 7857.51 examples/s]
Map (num_proc=16):  98%|█████████▊| 45418/46283 [00:09<00:00, 5239.35 examples/s]
Map (num_proc=16):  89%|████████▉ | 41364/46283 [00:07<00:00, 11281.03 examples/s]
                                                                                 

Map (num_proc=16):  92%|█████████▏| 42529/46283 [00:07<00:00, 10266.99 examples/s]
Map (num_proc=16):  99%|█████████▉| 46028/46283 [00:09<00:00, 3753.86 examples/s]
Map (num_proc=16):  94%|█████████▍| 43576/46283 [00:08<00:00, 9797.76 examples/s] 
Map (num_proc=16):  96%|█████████▋| 44607/46283 [00:08<00:00, 8476.63 examples/s]
                                                                                 

Map (num_proc=16):  98%|█████████▊| 45532/46283 [00:08<00:00, 7180.73 examples/s]
                                                                                 
Final fine-tuning data:
Train Dataset({
    features: ['input', 'prompt_type', 'source', 'input_ids', 'attention_mask', 'labels'],
    num_rows: 46283
})
Valid None
Sample input: {'input': ["<human>: scrivi una lettera di dimissioni da un'azienda informatica in cui hai lavorato 10 anni come ingegnere del software. Usa un tono formale\n<bot>: Egregio Direttore,\n\nSono con grande rimpianto che scrivo questa lettera di dimissioni dalla mia posizione di Ingegnere del Software presso la vostra azienda. Ho trascorso gli ultimi 10 anni presso questa azienda e sono stato onorato di far parte di un team così talentuoso e appassionato.\nTuttavia, ho deciso che è giunto il momento per me di intraprendere un nuovo cammino professionale e di esplorare nuove opportunità di crescita e sviluppo. Questa decisione non è stata facile da prendere, ma sono convinto che sia la cosa giusta da fare per il mio futuro.\n\nVorrei ringraziare tutti i miei colleghi per la collaborazione e il supporto che ho ricevuto durante il mio tempo presso la vostra azienda. In particolare, vorrei esprimere la mia gratitudine ai miei capi per avermi dato l'opportunità di lavorare su progetti interessanti e di sviluppare le mie capacità professionali.\n\nVi prego di accettare questa lettera come formalità di dimissioni dalla mia posizione a partire dalla prossima settimana. Sarò a vostra disposizione per supportare il processo di transizione e per rispondere a qualsiasi domanda possa avere durante questo periodo.\n\nGrazie ancora per la mia esperienza presso la vostra azienda e per tutte le opportunità che mi avete dato.\n\nDistinti saluti,\n[Il tuo nome]\n\n<human>: Grazie! Potresti modificare la bella mail che mi proponi di inviare al direttore, \n1. indicando che sono disponibile a negoziare un periodo di dimissioni in accordo con l'azienda, \n2. specificando che sono disponibile a valutare un'offerta di aumento di stipendio da parte dell'azienda, \nanche in relazione ai buoni risultati raggiunti nell'ultimo anno lavorativo.\n\n<bot>: Egregio Direttore,\n\nSono con grande rimpianto che scrivo questa lettera di dimissioni dalla mia posizione di Ingegnere del Software presso la vostra azienda. Ho trascorso gli ultimi 10 anni presso questa azienda e sono stato onorato di far parte di un team così talentuoso e appassionato. Tuttavia, ho deciso che è giunto il momento per me di intraprendere un nuovo cammino professionale e di esplorare nuove opportunità di crescita e sviluppo. Questa decisione non è stata facile da prendere, ma sono convinto che sia la cosa giusta da fare per il mio futuro.\n\nVorrei ringraziare tutti i miei colleghi per la collaborazione e il supporto che ho ricevuto durante il mio tempo presso la vostra azienda. In particolare, vorrei esprimere la mia gratitudine ai miei capi per avermi dato l'opportunità di lavorare su progetti interessanti e di sviluppare le mie capacità professionali.\n\nTengo anche a precisare che sono disponibile a negoziare un periodo di dimissioni in accordo con l'azienda e che sono pronto a valutare un'offerta di aumento di stipendio da parte dell'azienda, in riconoscimento dei buoni risultati raggiunti nell'ultimo anno lavorativo.\n\nVi prego di accettare questa lettera come formalità di dimissioni dalla mia posizione a partire dalla prossima settimana. Sarò a vostra disposizione per supportare il processo di transizione e per rispondere a qualsiasi domanda possa avere durante questo periodo.\n"], 'prompt_type': ['plain'], 'source': ['OpenAssistant/oasst1']}
Auto set eval_steps to 4 out of 90 total training steps
Auto step save_steps to 4

                                                                                 
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback
You are adding a <class 'transformers.integrations.TensorBoardCallback'> to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is
:DefaultFlowCallback
TensorBoardCallback

  0%|          | 0/90 [00:00<?, ?it/s]You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:219: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
  attn_scores = torch.where(causal_mask, attn_scores, mask_value)

  1%|          | 1/90 [01:55<2:51:41, 115.75s/it]
                                                 
{'loss': 1.6167, 'learning_rate': 2.9999999999999997e-06, 'epoch': 0.01}

  1%|          | 1/90 [01:55<2:51:41, 115.75s/it]
  2%|▏         | 2/90 [03:36<2:36:35, 106.76s/it]
                                                 
{'loss': 1.6205, 'learning_rate': 5.999999999999999e-06, 'epoch': 0.02}

  2%|▏         | 2/90 [03:36<2:36:35, 106.76s/it]
  3%|▎         | 3/90 [05:31<2:40:19, 110.56s/it]
                                                 
{'loss': 1.5936, 'learning_rate': 8.999999999999999e-06, 'epoch': 0.03}

  3%|▎         | 3/90 [05:31<2:40:19, 110.56s/it]
  4%|▍         | 4/90 [07:06<2:30:01, 104.66s/it]
                                                 
{'loss': 1.5738, 'learning_rate': 1.1999999999999999e-05, 'epoch': 0.04}

  4%|▍         | 4/90 [07:06<2:30:01, 104.66s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

  6%|▌         | 5/90 [08:48<2:26:30, 103.42s/it]
                                                 
{'loss': 1.5741, 'learning_rate': 1.4999999999999999e-05, 'epoch': 0.06}

  6%|▌         | 5/90 [08:48<2:26:30, 103.42s/it]
  7%|▋         | 6/90 [10:33<2:25:31, 103.95s/it]
                                                 
{'loss': 1.5732, 'learning_rate': 1.7999999999999997e-05, 'epoch': 0.07}

  7%|▋         | 6/90 [10:33<2:25:31, 103.95s/it]
  8%|▊         | 7/90 [12:29<2:29:36, 108.15s/it]
                                                 
{'loss': 1.5578, 'learning_rate': 2.1e-05, 'epoch': 0.08}

  8%|▊         | 7/90 [12:29<2:29:36, 108.15s/it]
  9%|▉         | 8/90 [14:14<2:26:18, 107.06s/it]
                                                 
{'loss': 1.533, 'learning_rate': 2.3999999999999997e-05, 'epoch': 0.09}

  9%|▉         | 8/90 [14:14<2:26:18, 107.06s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 10%|█         | 9/90 [16:04<2:25:37, 107.87s/it]
                                                 
{'loss': 1.5434, 'learning_rate': 2.6999999999999996e-05, 'epoch': 0.1}

 10%|█         | 9/90 [16:04<2:25:37, 107.87s/it]
 11%|█         | 10/90 [17:56<2:25:43, 109.29s/it]
                                                  
{'loss': 1.5204, 'learning_rate': 2.9999999999999997e-05, 'epoch': 0.11}

 11%|█         | 10/90 [17:56<2:25:43, 109.29s/it]
 12%|█▏        | 11/90 [19:57<2:28:23, 112.71s/it]
                                                  
{'loss': 1.5208, 'learning_rate': 3.2999999999999996e-05, 'epoch': 0.12}

 12%|█▏        | 11/90 [19:57<2:28:23, 112.71s/it]
 13%|█▎        | 12/90 [21:55<2:28:48, 114.47s/it]
                                                  
{'loss': 1.5555, 'learning_rate': 3.2999999999999996e-05, 'epoch': 0.13}

 13%|█▎        | 12/90 [21:55<2:28:48, 114.47s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 14%|█▍        | 13/90 [23:53<2:28:17, 115.56s/it]
                                                  
{'loss': 1.5096, 'learning_rate': 3.2999999999999996e-05, 'epoch': 0.14}

 14%|█▍        | 13/90 [23:53<2:28:17, 115.56s/it]
 16%|█▌        | 14/90 [25:34<2:20:37, 111.01s/it]
                                                  
{'loss': 1.5265, 'learning_rate': 3.5999999999999994e-05, 'epoch': 0.15}

 16%|█▌        | 14/90 [25:34<2:20:37, 111.01s/it]
 17%|█▋        | 15/90 [27:18<2:16:02, 108.83s/it]
                                                  
{'loss': 1.5215, 'learning_rate': 3.5999999999999994e-05, 'epoch': 0.17}

 17%|█▋        | 15/90 [27:18<2:16:02, 108.83s/it]
 18%|█▊        | 16/90 [29:02<2:12:33, 107.48s/it]
                                                  
{'loss': 1.5005, 'learning_rate': 3.9e-05, 'epoch': 0.18}

 18%|█▊        | 16/90 [29:02<2:12:33, 107.48s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 19%|█▉        | 17/90 [30:49<2:10:26, 107.22s/it]
                                                  
{'loss': 1.5395, 'learning_rate': 4.2e-05, 'epoch': 0.19}

 19%|█▉        | 17/90 [30:49<2:10:26, 107.22s/it]
 20%|██        | 18/90 [32:21<2:03:29, 102.90s/it]
                                                  
{'loss': 1.5301, 'learning_rate': 4.4999999999999996e-05, 'epoch': 0.2}

 20%|██        | 18/90 [32:21<2:03:29, 102.90s/it]
 21%|██        | 19/90 [33:59<1:59:49, 101.25s/it]
                                                  
{'loss': 1.5227, 'learning_rate': 4.7999999999999994e-05, 'epoch': 0.21}

 21%|██        | 19/90 [33:59<1:59:49, 101.25s/it]
 22%|██▏       | 20/90 [35:50<2:01:36, 104.24s/it]
                                                  
{'loss': 1.5507, 'learning_rate': 4.7999999999999994e-05, 'epoch': 0.22}

 22%|██▏       | 20/90 [35:50<2:01:36, 104.24s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 23%|██▎       | 21/90 [37:33<1:59:33, 103.96s/it]
                                                  
{'loss': 1.5371, 'learning_rate': 5.1e-05, 'epoch': 0.23}

 23%|██▎       | 21/90 [37:33<1:59:33, 103.96s/it]
 24%|██▍       | 22/90 [39:22<1:59:28, 105.41s/it]
                                                  
{'loss': 1.5185, 'learning_rate': 5.399999999999999e-05, 'epoch': 0.24}

 24%|██▍       | 22/90 [39:22<1:59:28, 105.41s/it]
 26%|██▌       | 23/90 [41:07<1:57:41, 105.39s/it]
                                                  
{'loss': 1.5261, 'learning_rate': 5.6999999999999996e-05, 'epoch': 0.25}

 26%|██▌       | 23/90 [41:07<1:57:41, 105.39s/it]
 27%|██▋       | 24/90 [42:37<1:50:40, 100.61s/it]
                                                  
{'loss': 1.5248, 'learning_rate': 5.6999999999999996e-05, 'epoch': 0.27}

 27%|██▋       | 24/90 [42:37<1:50:40, 100.61s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 28%|██▊       | 25/90 [44:16<1:48:21, 100.02s/it]
                                                  
{'loss': 1.5126, 'learning_rate': 5.9999999999999995e-05, 'epoch': 0.28}

 28%|██▊       | 25/90 [44:16<1:48:21, 100.02s/it]
 29%|██▉       | 26/90 [46:04<1:49:22, 102.53s/it]
                                                  
{'loss': 1.4811, 'learning_rate': 6.299999999999999e-05, 'epoch': 0.29}

 29%|██▉       | 26/90 [46:04<1:49:22, 102.53s/it]
 30%|███       | 27/90 [47:48<1:48:17, 103.13s/it]
                                                  
{'loss': 1.52, 'learning_rate': 6.599999999999999e-05, 'epoch': 0.3}

 30%|███       | 27/90 [47:49<1:48:17, 103.13s/it]
 31%|███       | 28/90 [49:29<1:45:53, 102.47s/it]
                                                  
{'loss': 1.4843, 'learning_rate': 6.9e-05, 'epoch': 0.31}

 31%|███       | 28/90 [49:29<1:45:53, 102.47s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 32%|███▏      | 29/90 [51:18<1:45:57, 104.22s/it]
                                                  
{'loss': 1.4755, 'learning_rate': 7.199999999999999e-05, 'epoch': 0.32}

 32%|███▏      | 29/90 [51:18<1:45:57, 104.22s/it]
 33%|███▎      | 30/90 [53:04<1:44:47, 104.80s/it]
                                                  
{'loss': 1.4695, 'learning_rate': 7.5e-05, 'epoch': 0.33}

 33%|███▎      | 30/90 [53:04<1:44:47, 104.80s/it]
 34%|███▍      | 31/90 [54:52<1:44:04, 105.85s/it]
                                                  
{'loss': 1.5426, 'learning_rate': 7.8e-05, 'epoch': 0.34}

 34%|███▍      | 31/90 [54:52<1:44:04, 105.85s/it]
 36%|███▌      | 32/90 [56:33<1:40:47, 104.27s/it]
                                                  
{'loss': 1.498, 'learning_rate': 8.1e-05, 'epoch': 0.35}

 36%|███▌      | 32/90 [56:33<1:40:47, 104.27s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 37%|███▋      | 33/90 [58:15<1:38:33, 103.74s/it]
                                                  
{'loss': 1.5435, 'learning_rate': 8.4e-05, 'epoch': 0.36}

 37%|███▋      | 33/90 [58:15<1:38:33, 103.74s/it]
 38%|███▊      | 34/90 [59:58<1:36:30, 103.40s/it]
                                                  
{'loss': 1.4837, 'learning_rate': 8.699999999999999e-05, 'epoch': 0.38}

 38%|███▊      | 34/90 [59:58<1:36:30, 103.40s/it]
 39%|███▉      | 35/90 [1:01:40<1:34:29, 103.08s/it]
                                                    
{'loss': 1.4846, 'learning_rate': 8.999999999999999e-05, 'epoch': 0.39}

 39%|███▉      | 35/90 [1:01:40<1:34:29, 103.08s/it]
 40%|████      | 36/90 [1:03:30<1:34:37, 105.15s/it]
                                                    
{'loss': 1.5164, 'learning_rate': 9.3e-05, 'epoch': 0.4}

 40%|████      | 36/90 [1:03:30<1:34:37, 105.15s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 41%|████      | 37/90 [1:05:28<1:36:22, 109.11s/it]
                                                    
{'loss': 1.4747, 'learning_rate': 9.599999999999999e-05, 'epoch': 0.41}

 41%|████      | 37/90 [1:05:28<1:36:22, 109.11s/it]
 42%|████▏     | 38/90 [1:07:29<1:37:28, 112.46s/it]
                                                    
{'loss': 1.4525, 'learning_rate': 9.9e-05, 'epoch': 0.42}

 42%|████▏     | 38/90 [1:07:29<1:37:28, 112.46s/it]
 43%|████▎     | 39/90 [1:09:09<1:32:27, 108.77s/it]
                                                    
{'loss': 1.4976, 'learning_rate': 0.000102, 'epoch': 0.43}

 43%|████▎     | 39/90 [1:09:09<1:32:27, 108.77s/it]
 44%|████▍     | 40/90 [1:11:03<1:32:02, 110.45s/it]
                                                    
{'loss': 1.5076, 'learning_rate': 0.00010499999999999999, 'epoch': 0.44}

 44%|████▍     | 40/90 [1:11:03<1:32:02, 110.45s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 46%|████▌     | 41/90 [1:12:50<1:29:23, 109.46s/it]
                                                    
{'loss': 1.4636, 'learning_rate': 0.00010799999999999998, 'epoch': 0.45}

 46%|████▌     | 41/90 [1:12:50<1:29:23, 109.46s/it]
 47%|████▋     | 42/90 [1:14:42<1:27:58, 109.96s/it]
                                                    
{'loss': 1.4405, 'learning_rate': 0.00011099999999999999, 'epoch': 0.46}

 47%|████▋     | 42/90 [1:14:42<1:27:58, 109.96s/it]
 48%|████▊     | 43/90 [1:16:30<1:25:43, 109.43s/it]
                                                    
{'loss': 1.4986, 'learning_rate': 0.00011399999999999999, 'epoch': 0.48}

 48%|████▊     | 43/90 [1:16:30<1:25:43, 109.43s/it]
 49%|████▉     | 44/90 [1:18:34<1:27:21, 113.96s/it]
                                                    
{'loss': 1.4735, 'learning_rate': 0.000117, 'epoch': 0.49}

 49%|████▉     | 44/90 [1:18:34<1:27:21, 113.96s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 50%|█████     | 45/90 [1:20:31<1:25:58, 114.64s/it]
                                                    
{'loss': 1.4659, 'learning_rate': 0.00011999999999999999, 'epoch': 0.5}

 50%|█████     | 45/90 [1:20:31<1:25:58, 114.64s/it]
 51%|█████     | 46/90 [1:22:20<1:23:01, 113.21s/it]
                                                    
{'loss': 1.4485, 'learning_rate': 0.00012299999999999998, 'epoch': 0.51}

 51%|█████     | 46/90 [1:22:20<1:23:01, 113.21s/it]
 52%|█████▏    | 47/90 [1:23:57<1:17:36, 108.29s/it]
                                                    
{'loss': 1.5387, 'learning_rate': 0.00012599999999999997, 'epoch': 0.52}

 52%|█████▏    | 47/90 [1:23:57<1:17:36, 108.29s/it]
 53%|█████▎    | 48/90 [1:25:57<1:18:16, 111.83s/it]
                                                    
{'loss': 1.4543, 'learning_rate': 0.000129, 'epoch': 0.53}

 53%|█████▎    | 48/90 [1:25:57<1:18:16, 111.83s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 54%|█████▍    | 49/90 [1:27:51<1:16:45, 112.33s/it]
                                                    
{'loss': 1.5208, 'learning_rate': 0.00013199999999999998, 'epoch': 0.54}

 54%|█████▍    | 49/90 [1:27:51<1:16:45, 112.33s/it]
 56%|█████▌    | 50/90 [1:29:41<1:14:25, 111.63s/it]
                                                    
{'loss': 1.4878, 'learning_rate': 0.000135, 'epoch': 0.55}

 56%|█████▌    | 50/90 [1:29:41<1:14:25, 111.63s/it]
 57%|█████▋    | 51/90 [1:31:29<1:11:49, 110.51s/it]
                                                    
{'loss': 1.4765, 'learning_rate': 0.000138, 'epoch': 0.56}

 57%|█████▋    | 51/90 [1:31:29<1:11:49, 110.51s/it]
 58%|█████▊    | 52/90 [1:33:21<1:10:22, 111.13s/it]
                                                    
{'loss': 1.4426, 'learning_rate': 0.00014099999999999998, 'epoch': 0.57}

 58%|█████▊    | 52/90 [1:33:21<1:10:22, 111.13s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 59%|█████▉    | 53/90 [1:35:05<1:07:12, 108.99s/it]
                                                    
{'loss': 1.4804, 'learning_rate': 0.00014399999999999998, 'epoch': 0.59}

 59%|█████▉    | 53/90 [1:35:05<1:07:12, 108.99s/it]
 60%|██████    | 54/90 [1:36:56<1:05:39, 109.44s/it]
                                                    
{'loss': 1.4326, 'learning_rate': 0.000147, 'epoch': 0.6}

 60%|██████    | 54/90 [1:36:56<1:05:39, 109.44s/it]
 61%|██████    | 55/90 [1:38:28<1:00:48, 104.26s/it]
                                                    
{'loss': 1.4491, 'learning_rate': 0.00015, 'epoch': 0.61}

 61%|██████    | 55/90 [1:38:28<1:00:48, 104.26s/it]
 62%|██████▏   | 56/90 [1:40:13<59:14, 104.55s/it]  
                                                  
{'loss': 1.4539, 'learning_rate': 0.00015299999999999998, 'epoch': 0.62}

 62%|██████▏   | 56/90 [1:40:13<59:14, 104.55s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 63%|██████▎   | 57/90 [1:42:04<58:31, 106.40s/it]
                                                  
{'loss': 1.4549, 'learning_rate': 0.000156, 'epoch': 0.63}

 63%|██████▎   | 57/90 [1:42:04<58:31, 106.40s/it]
 64%|██████▍   | 58/90 [1:43:51<56:52, 106.64s/it]
                                                  
{'loss': 1.433, 'learning_rate': 0.000159, 'epoch': 0.64}

 64%|██████▍   | 58/90 [1:43:51<56:52, 106.64s/it]
 66%|██████▌   | 59/90 [1:45:38<55:09, 106.76s/it]
                                                  
{'loss': 1.481, 'learning_rate': 0.000162, 'epoch': 0.65}

 66%|██████▌   | 59/90 [1:45:38<55:09, 106.76s/it]
 67%|██████▋   | 60/90 [1:47:25<53:20, 106.69s/it]
                                                  
{'loss': 1.4615, 'learning_rate': 0.000165, 'epoch': 0.66}

 67%|██████▋   | 60/90 [1:47:25<53:20, 106.69s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 68%|██████▊   | 61/90 [1:48:55<49:15, 101.91s/it]
                                                  
{'loss': 1.4499, 'learning_rate': 0.000168, 'epoch': 0.67}

 68%|██████▊   | 61/90 [1:48:55<49:15, 101.91s/it]
 69%|██████▉   | 62/90 [1:50:42<48:14, 103.39s/it]
                                                  
{'loss': 1.4404, 'learning_rate': 0.00017099999999999998, 'epoch': 0.69}

 69%|██████▉   | 62/90 [1:50:42<48:14, 103.39s/it]
 70%|███████   | 63/90 [1:52:39<48:21, 107.45s/it]
                                                  
{'loss': 1.4656, 'learning_rate': 0.00017399999999999997, 'epoch': 0.7}

 70%|███████   | 63/90 [1:52:39<48:21, 107.45s/it]
 71%|███████   | 64/90 [1:54:31<47:07, 108.73s/it]
                                                  
{'loss': 1.3841, 'learning_rate': 0.00017699999999999997, 'epoch': 0.71}

 71%|███████   | 64/90 [1:54:31<47:07, 108.73s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 72%|███████▏  | 65/90 [1:56:20<45:17, 108.71s/it]
                                                  
{'loss': 1.4493, 'learning_rate': 0.00017999999999999998, 'epoch': 0.72}

 72%|███████▏  | 65/90 [1:56:20<45:17, 108.71s/it]
 73%|███████▎  | 66/90 [1:58:05<43:05, 107.73s/it]
                                                  
{'loss': 1.4808, 'learning_rate': 0.00018299999999999998, 'epoch': 0.73}

 73%|███████▎  | 66/90 [1:58:05<43:05, 107.73s/it]
 74%|███████▍  | 67/90 [1:59:59<42:01, 109.64s/it]
                                                  
{'loss': 1.4782, 'learning_rate': 0.000186, 'epoch': 0.74}

 74%|███████▍  | 67/90 [1:59:59<42:01, 109.64s/it]
 76%|███████▌  | 68/90 [2:01:47<40:03, 109.23s/it]
                                                  
{'loss': 1.4596, 'learning_rate': 0.00018899999999999999, 'epoch': 0.75}

 76%|███████▌  | 68/90 [2:01:47<40:03, 109.23s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 77%|███████▋  | 69/90 [2:03:34<37:56, 108.42s/it]
                                                  
{'loss': 1.5132, 'learning_rate': 0.00019199999999999998, 'epoch': 0.76}

 77%|███████▋  | 69/90 [2:03:34<37:56, 108.42s/it]
 78%|███████▊  | 70/90 [2:05:08<34:41, 104.06s/it]
                                                  
{'loss': 1.4905, 'learning_rate': 0.000195, 'epoch': 0.77}

 78%|███████▊  | 70/90 [2:05:08<34:41, 104.06s/it]
 79%|███████▉  | 71/90 [2:06:59<33:39, 106.29s/it]
                                                  
{'loss': 1.4231, 'learning_rate': 0.000198, 'epoch': 0.79}

 79%|███████▉  | 71/90 [2:06:59<33:39, 106.29s/it]
 80%|████████  | 72/90 [2:08:40<31:25, 104.76s/it]
                                                  
{'loss': 1.488, 'learning_rate': 0.000201, 'epoch': 0.8}

 80%|████████  | 72/90 [2:08:40<31:25, 104.76s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 81%|████████  | 73/90 [2:10:20<29:12, 103.08s/it]
                                                  
{'loss': 1.4238, 'learning_rate': 0.000204, 'epoch': 0.81}

 81%|████████  | 73/90 [2:10:20<29:12, 103.08s/it]
 82%|████████▏ | 74/90 [2:12:17<28:36, 107.26s/it]
                                                  
{'loss': 1.4786, 'learning_rate': 0.00020699999999999996, 'epoch': 0.82}

 82%|████████▏ | 74/90 [2:12:17<28:36, 107.26s/it]
 83%|████████▎ | 75/90 [2:14:18<27:52, 111.48s/it]
                                                  
{'loss': 1.5053, 'learning_rate': 0.00020999999999999998, 'epoch': 0.83}

 83%|████████▎ | 75/90 [2:14:18<27:52, 111.48s/it]
 84%|████████▍ | 76/90 [2:16:03<25:34, 109.64s/it]
                                                  
{'loss': 1.4512, 'learning_rate': 0.00021299999999999997, 'epoch': 0.84}

 84%|████████▍ | 76/90 [2:16:03<25:34, 109.64s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 86%|████████▌ | 77/90 [2:17:46<23:17, 107.52s/it]
                                                  
{'loss': 1.4497, 'learning_rate': 0.00021599999999999996, 'epoch': 0.85}

 86%|████████▌ | 77/90 [2:17:46<23:17, 107.52s/it]
 87%|████████▋ | 78/90 [2:19:58<22:59, 114.94s/it]
                                                  
{'loss': 1.3996, 'learning_rate': 0.00021899999999999998, 'epoch': 0.86}

 87%|████████▋ | 78/90 [2:19:58<22:59, 114.94s/it]
 88%|████████▊ | 79/90 [2:21:56<21:13, 115.77s/it]
                                                  
{'loss': 1.4669, 'learning_rate': 0.00022199999999999998, 'epoch': 0.87}

 88%|████████▊ | 79/90 [2:21:56<21:13, 115.77s/it]
 89%|████████▉ | 80/90 [2:23:39<18:40, 112.04s/it]
                                                  
{'loss': 1.4266, 'learning_rate': 0.000225, 'epoch': 0.88}

 89%|████████▉ | 80/90 [2:23:39<18:40, 112.04s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 90%|█████████ | 81/90 [2:25:14<16:01, 106.84s/it]
                                                  
{'loss': 1.4923, 'learning_rate': 0.00022799999999999999, 'epoch': 0.9}

 90%|█████████ | 81/90 [2:25:14<16:01, 106.84s/it]
 91%|█████████ | 82/90 [2:26:59<14:11, 106.41s/it]
                                                  
{'loss': 1.4442, 'learning_rate': 0.00023099999999999998, 'epoch': 0.91}

 91%|█████████ | 82/90 [2:26:59<14:11, 106.41s/it]
 92%|█████████▏| 83/90 [2:28:44<12:22, 106.03s/it]
                                                  
{'loss': 1.4751, 'learning_rate': 0.000234, 'epoch': 0.92}

 92%|█████████▏| 83/90 [2:28:44<12:22, 106.03s/it]
 93%|█████████▎| 84/90 [2:30:20<10:18, 103.00s/it]
                                                  
{'loss': 1.4661, 'learning_rate': 0.000237, 'epoch': 0.93}

 93%|█████████▎| 84/90 [2:30:20<10:18, 103.00s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 94%|█████████▍| 85/90 [2:32:20<09:00, 108.05s/it]
                                                  
{'loss': 1.4366, 'learning_rate': 0.00023999999999999998, 'epoch': 0.94}

 94%|█████████▍| 85/90 [2:32:20<09:00, 108.05s/it]
 96%|█████████▌| 86/90 [2:34:00<07:02, 105.57s/it]
                                                  
{'loss': 1.4952, 'learning_rate': 0.000243, 'epoch': 0.95}

 96%|█████████▌| 86/90 [2:34:00<07:02, 105.57s/it]
 97%|█████████▋| 87/90 [2:35:52<05:22, 107.55s/it]
                                                  
{'loss': 1.4255, 'learning_rate': 0.00024599999999999996, 'epoch': 0.96}

 97%|█████████▋| 87/90 [2:35:52<05:22, 107.55s/it]
 98%|█████████▊| 88/90 [2:37:32<03:30, 105.14s/it]
                                                  
{'loss': 1.4955, 'learning_rate': 0.000249, 'epoch': 0.97}

 98%|█████████▊| 88/90 [2:37:32<03:30, 105.14s/it]/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/bitsandbytes-0.38.0.post2-py3.10.egg/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

 99%|█████████▉| 89/90 [2:39:16<01:45, 105.03s/it]
                                                  
{'loss': 1.4546, 'learning_rate': 0.00025199999999999995, 'epoch': 0.98}

 99%|█████████▉| 89/90 [2:39:16<01:45, 105.03s/it]
100%|██████████| 90/90 [2:40:57<00:00, 103.73s/it]
                                                  
{'loss': 1.4536, 'learning_rate': 0.00025499999999999996, 'epoch': 1.0}

100%|██████████| 90/90 [2:40:57<00:00, 103.73s/it]
                                                  
{'train_runtime': 9671.9012, 'train_samples_per_second': 4.785, 'train_steps_per_second': 0.009, 'train_loss': 1.4906181216239929, 'epoch': 1.0}

100%|██████████| 90/90 [2:40:57<00:00, 103.73s/it]
100%|██████████| 90/90 [2:40:57<00:00, 107.31s/it]

 If there's a warning about missing keys above, please disregard :)
