Fd dZddlmZddlmZddlZddlZddlmZddl m Z ddl m Z ddl mZmZdd lmZdd lmZdd ed eed efdZ ddededeed efdZ ddededeefdZ ddededeedeed eedeeeefzff dZ ddedededeefdZdS)zText processing functions)ceil)OptionalN)Config) ChatSequence)OPEN_AI_MODELS)count_string_tokenscreate_chat_completion)logger)batchmodelmaxreturnc`t|jdz }||dkrt||S|S)Nr)r max_tokensmin)r r model_max_input_tokenss KC:\Users\Administrator\Downloads\Auto-GPT-master\autogpt\processing\text.py_max_chunk_lengthrs9+E2=A 3773./// !!text for_modelmax_chunk_lengthcFt||t||kSN)rr)rrrs rmust_chunk_contentrs. tY / /2C#33 rTcontentc#Kd}t|||s|t||fVdS|pt|}tj|}||}t |}t||z }t||z } |rt|| z |nd} t|| | z| D](} | | t | fV)dS)z>Split content into chunks of approximately equal token length.Nr) rrrtiktokenencoding_for_modelencodelenrrr decode) rrr with_overlap MAX_OVERLAP tokenizertokenized_text total_lengthn_chunks chunk_lengthoverlap token_batchs r chunk_contentr.s K gy2B C C*7I>>>>>>'G+> {++S-=-======>>rconfig instructionquestionc  |std|r|rtd|j}|rd|d}tj|}t ||}t jd|dt|dz }t jd|dt|||s| d d |d |nd d|dt j d|d| dt||ddj }t j ddddd|ddd |d fSgtt!|||| t# D]^\} \} } t jd| dzdt% d| dt'| |\}} |_t jdt% d t'd!\}} | fd"t-dt% DfS)#aSummarize text using the OpenAI API Args: text (str): The text to summarize config (Config): The config object instruction (str): Additional instruction for summarization, e.g. "focus on information related to polar bears", "omit personal information contained in the text" question (str): Question to answer in the summary Returns: str: The summary of the text list[(summary, chunk)]: Text chunks and their summary, if the text was chunked. None otherwise. zNo text to summarizez;Parameters 'question' and 'instructions' cannot both be setzAinclude any information that can be used to answer the question "z-". Do not directly answer the question itselfz Text length: z tokensi&zMax chunk length: userz-Write a concise summary of the following textNz; z: LITERAL TEXT: """z5""" CONCISE SUMMARY: The text is best summarized aszSummarizing with z:  ri)promptr/ temperaturerz----------------z SUMMARY z-----------------z*------------------------------------------)rr/rzSummarizing chunk rz / z of length z Summarized z chunksz c<g|]}||dfS)r).0ichunks summariess r z"summarize_text..s7)*1vay|$r) ValueErrorfast_llm_modelrrrr inforradddebugdumpr rstriplist split_text enumerater#summarize_textappendjoinrange)rr/r0r1r summarization_prompt token_lengthrsummaryr;chunkr+_r<r=s @@rrIrI<s,& 1/000XxXVWWW  !E  9PX 9 9 9  (1%88&tU33L K5 555666)//#5 K>%5>>>??? dE+; < <%   >%0%<!K!!!" > >!% > > >   RRR3G3L3L3N3NRRRSSS('ARU      J&JJ6JJWJJJJJKKK}}$$I  E&CS   F %.f$5$5""  E< XQ X X3v;; X X< X X X   $E;77 !!!! K2c&kk222333 I 6 677JGQ ==??.3As6{{.C.C rc#,Kt||}|dd}t||}||kr||fVdSt||z }t||z }t j|j} | d| |} d| jD} g} d} d}d}d}|t| kr$| |}t||}| dz|z}||kr$||dz z |kr| ||} n||kr|rd | | fVg} d} |re||z dz }||kr| |gz } | |dzz } nH|d krB| tt|||dgz } | |dzz } | |gz } | |z } n%d t|||D| ||dz<|dz }|}|}|t| k$| rd | | fVdSdS) a'Split text into chunks of sentences, with each chunk not exceeding the maximum length Args: text (str): The text to split for_model (str): The model to chunk for; determines tokenizer and constraints config (Config): The config object with_overlap (bool, optional): Whether to allow overlap between chunks max_chunk_length (int, optional): The maximum length of a chunk Yields: str: The next chunk of text Raises: ValueError: when a sentence is longer than the maximum length r5 N sentencizerc@g|]}|jSr9)rrE)r:sentences rr>zsplit_text..s&AAA8$$&&AAArrrcg|]\}}|Sr9r9)r:rPrQs rr>zsplit_text..s,$$$E1$$$r)rreplacerrspacyloadbrowse_spacy_language_modeladd_pipesentsr#rJrKrFr.pop)rrr/r%r max_length text_lengthr*target_chunk_lengthnlpdoc sentences current_chunkcurrent_chunk_length last_sentencelast_sentence_lengthr;rVsentence_lengthexpected_chunk_lengthoverlap_max_lengths rrGrGs.#9.>??J <<c " "D%dI66KZKK*,--H{X566#(:f.P#Q#QCLL #d))CAAsyAAAI!MM A c)nn  Q<-h BB 4q 8? J "J . .%1)<=@SSS   * * *#8 z ) ) Ghh}--/CCCCC " '($G)3o)E)I&+.@@@%-8 ,0Dq0HH,,+a//% -$1$-$6!"!" "ceeA '* -0BQ0FF, hZ 'M O 3 $$ -h CV W W$$$Ia!a%i   Q .c c)nn  f<hh}%%';;;;;;;<r{sE !!!!!!))))))777777IIIIIIII""S"x}"""""BF 19# '+ >> >>sm>>>>@"&" SS S S#Ssm S  3tE#s(O,, ,- SSSSt&* c<c< c<c< c< sm c<c<c<c<c<c