sg+ d dl Z d dlmZ d dlmZmZ d dlmZ ddlm Z ddl
m Z m Z ddl
mZ ed e rd
d lmZ ndZ e j" e Zd d
dZd\ ZZd\ ZZdZ G d de Zy) N)copyfile)OptionalTuple)
processors )PreTrainedTokenizerFast)is_sentencepiece_availablelogging)require_versionztokenizers>=0.13.3 )LlamaTokenizerztokenizer.modelztokenizer.json)
vocab_filetokenizer_file)z[INST]z[/INST])z<>
z
<>
a You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.c e Zd ZdZeZeZdZddgZ d fd Z
e de fd Z
d Ze d Ze d
Zej" d Zej" d Zdd
edee dee fdZddZ xZS )LlamaTokenizerFastu
Construct a Llama tokenizer. Based on byte-level Byte-Pair-Encoding.
This uses notably ByteFallback and no normalization.
```python
>>> from transformers import LlamaTokenizerFast
>>> tokenizer = LlamaTokenizerFast.from_pretrained("hf-internal-testing/llama-tokenizer")
>>> tokenizer.encode("Hello this is a test")
[1, 15043, 445, 338, 263, 1243]
```
If you want to change the `bos_token` or the `eos_token`, make sure to specify them when initializing the model, or
call `tokenizer.update_post_processor()` to make sure that the post-processing is correctly done (otherwise the
values of the first token and final token of an encoded sequence will not be correct). For more details, checkout
[post-processors] (https://huggingface.co/docs/tokenizers/api/post-processors) documentation.
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`, *optional*):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a .model extension) that
contains the vocabulary necessary to instantiate a tokenizer.
tokenizer_file (`str`, *optional*):
[tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that
contains everything needed to load the tokenizer.
clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like
extra spaces.
unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`):
The end of sequence token.
add_bos_token (`bool`, *optional*, defaults to `True`):
Whether or not to add an `bos_token` at the start of sequences.
add_eos_token (`bool`, *optional*, defaults to `False`):
Whether or not to add an `eos_token` at the end of sequences.
use_default_system_prompt (`bool`, *optional*, defaults to `False`):
Whether or not the default system prompt for Llama should be used
legacy (`bool`, *optional*):
Whether or not the `legacy` behavior of the tokenizer should be used. Legacy is before the merge of #24622
and #25224 which includes fixes to properly handle tokens that appear after special tokens.
Make sure to also set `from_slow` to `True`.
A simple example:
- `legacy=True`:
```python
>>> from transformers import LlamaTokenizerFast
>>> tokenizer = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", legacy=True, from_slow=True)
>>> tokenizer.encode("Hello .") # 869 is '▁.'
[1, 15043, 29871, 1, 869]
```
- `legacy=False`:
```python
>>> from transformers import LlamaTokenizerFast
>>> tokenizer = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", legacy=False, from_slow=True)
>>> tokenizer.encode("Hello .") # 29889 is '.'
[1, 15043, 29871, 1, 29889]
```
Checkout the [pull request](https://github.com/huggingface/transformers/pull/24565) for more details.
add_prefix_space (`bool`, *optional*):
Whether or not the tokenizer should automatically add a prefix space
left input_idsattention_maskc |
%t j d| j d d}
|
| _ | d| d< t
| d||||||||| | |
d | || _ || _ | j | | _ || _
y )Nz2You are using the default legacy behaviour of the a . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.T from_slow) r r clean_up_tokenization_spaces unk_token bos_token eos_token
add_bos_token
add_eos_tokenuse_default_system_promptadd_prefix_spacelegacy ) logger warning_once __class__r super__init___add_bos_token_add_eos_tokenupdate_post_processorr r )selfr r r r r r r r r r r kwargsr# s d/var/www/html/venv/lib/python3.12/site-packages/transformers/models/llama/tokenization_llama_fast.pyr% zLlamaTokenizerFast.__init__ s > DT^^DT U0 0
F '"&F;
!))E ''&? -
, + ""$)B&$ returnc p | j r)t j j | j S dS )NF)r ospathisfiler) s r+ can_save_slow_tokenizerz*LlamaTokenizerFast.can_save_slow_tokenizer s$ 26//rww~~doo.LuLr, c P | j }| j }|| j r t d | j }| j
}|| j r t d | j r|dz nd d| j rd|z dz nd }| | j rd|z d z nd d
| j rd|z d z nd }g }| j r|j ||f | j r|j ||f t j ||| | j _ y) ze
Updates the underlying post processor with the current `bos_token` and `eos_token`.
Nz)add_bos_token = True but bos_token = Nonez)add_eos_token = True but eos_token = Nonez:0 z$A:0 z:0z:1z $B:1)singlepairspecial_tokens) r bos_token_idr
ValueErrorr eos_token_idr appendr TemplateProcessing
_tokenizerpost_processor)r) bosr: eosr< r7 r8 r9 s r+ r( z(LlamaTokenizerFast.update_post_processor s+ nn(( ;4-- HI Inn(( ;4-- HI I#'#5#5SY2>dUYUgUgCGDLmoBpqD,>,>3s74zzTFFNNrD )__name__
__module__ __qualname____doc__rS vocab_files_namesr
slow_tokenizer_class padding_sidemodel_input_namesr% propertyboolr3 r( r r setterstrr r rV r[
__classcell__)r# s @r+ r r 1 s FP *)L$&67 %*"' 0%d M M M
4 # # # # % % % %!c !HSM !]bcf]g !* r, r )r/ shutilr typingr r
tokenizersr tokenization_utils_fastr utilsr r
utils.versionsr tokenization_llamar
get_loggerr\ r! rS B_INSTE_INSTB_SYSE_SYSDEFAULT_SYSTEM_PROMPTr r r, r+ rv s}
" ! > 8 - $ % 2N H %#4HXY $, u^ N0 Nr,