add-determinism 0.7.3

3

�
�^=mc@s<dZdZddlZddlZddlZddlZddlZddlZddlm	Z	ddl
mZmZm
Z
ddlmZddlmZmZdd	lmZd&dd�ZGd
d�de�Zdd�Zdd�Zdd�Zdd�Zejd'kr�ddlZejZdd�Z dd�Z!neZeZ eZ!dd�Z"Gdd�de�Z#Gd d!�d!e$�Z%Gd"d#�d#e�Z&Gd$d%�d%e%�Z'dS)(z�Refactoring framework.

Used as a main program, this can refactor any number of files and/or
recursively descend down directories.  Imported as a module, this
provides infrastructure to write your own refactoring tool.
z#Guido van Rossum <guido@python.org>�N)�chain�)�driver�tokenize�token)�	find_root)�pytree�pygram)�btm_matcherTcCstt|ggdg�}tjj|j�}g}xLttj|��D]:}|jd�r2|jd�r2|rZ|dd�}|j	|dd��q2W|S)zEReturn a sorted list of all available fix names in the given package.�*�fix_z.pyiN�i����)
�
__import__�os�pathZdirnameZ__file__ZsortedZlistdir�
startswith�endswith�append)Z	fixer_pkgZ
remove_prefixZpkgZ	fixer_dirZ	fix_names�name�r�(/usr/lib64/python3.6/lib2to3/refactor.py�get_all_fix_namessrc@�eZdZdS)�
_EveryNodeN��__name__�
__module__�__qualname__rrrrr+�rcCs�t|tjtjf�r(|jdkr t�|jhSt|tj�rH|jrDt|j�St�t|tj	�r�t
�}x*|jD] }x|D]}|jt|��qlWqbW|Std|��dS)zf Accepts a pytree Pattern Node and returns a set
        of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s)
Z
isinstancerZNodePatternZLeafPattern�typerZNegatedPatternZcontent�_get_head_typesZWildcardPattern�set�update�	Exception)Zpat�r�pZxrrrr/s


rcCs�tjt�}g}x�|D]|}|jrjyt|j�}Wntk
rJ|j|�Yq�XxB|D]}||j|�qRWq|jdk	r�||jj|�q|j|�qWx,tt	j
jj�t	j
j
�D]}||j|�q�Wt|�S)z^ Accepts a list of fixers and returns a dictionary
        of head node type --> fixer list.  N)�collectionsZdefaultdict�listZpatternrrrZ_accept_typerr�python_grammarZ
symbol2number�values�tokens�extendZdict)Z
fixer_listZ
head_nodesZevery�fixerZheadsZ	node_typerrr�_get_headnode_dictKs"



r,cs�fdd�t�d�D�S)zN
    Return the fully qualified names for fixers in the package pkg_name.
    csg|]}�d|�qS��.r)�.0�fix_name�Zpkg_namerr�
<listcomp>h�z+get_fixers_from_package.<locals>.<listcomp>F)rr1rr1r�get_fixers_from_packageds
r4cCs|S�Nr)Zobjrrr�	_identityk�r6rcCs|jdd�S)Nz
�
)�replace��inputrrr�_from_system_newlinesrr7r<cCs tjdkr|jdtj�S|SdS)Nr8)rZlinesepr9r:rrr�_to_system_newlinests
r=csTd}tjtj|�j���fdd�}ttjtjtj	h�}t
�}y�x�|�\}}||krVq@q@|tjkrl|rfPd}q@|tjko||dk�r,|�\}}|tjks�|dkr�P|�\}}|tjks�|dkr�P|�\}}|tj
kr�|dkr�|�\}}xJ|tjk�r(|j|�|�\}}|tj
k�s|d	k�rP|�\}}q�Wq@Pq@WWntk
�rJYnXt|�S)
NFcst��}|d|dfS)Nrr)Znext)Ztok�Zgenrr�advance�sz(_detect_future_features.<locals>.advanceTZfromZ
__future__Zimportz(z,)r�generate_tokens�ioZStringIO�readline�	frozensetrZNEWLINEZNLZCOMMENTr ZSTRINGZNAMEZOPZaddZ
StopIteration)ZsourceZhave_docstringr?Zignore�featuresZtp�valuerr>r�_detect_future_featuressD








rFc@seZdZdZdS)�
FixerErrorzA fixer could not be loaded.N)rrr�__doc__rrrrrG�rrGc@s�eZdZddd�ZdZdZd4dd�Zdd	�Zd
d�Zdd
�Z	dd�Z
dd�Zd5dd�Zd6dd�Z
dd�Zd7dd�Zdd�Zd8dd�Zdd�Zd d!�Zd9d"d#�Zd:d$d%�Zd&Zd'Zd(d)�Zd*d+�Zd,d-�Zd.d/�Zd0d1�Zd2d3�ZdS);�RefactoringToolF)�print_function�write_unchanged_filesZFixrNcCs2||_|pg|_|jj�|_|dk	r0|jj|�|jdrDtj|_ntj	|_|jj
d�|_g|_t
jd�|_g|_d|_tj|jtj|jd�|_|j�\|_|_g|_tj�|_g|_g|_xXt|j|j�D]F}|j r�|jj!|�q�||jkr�|jj"|�q�||jkr�|jj"|�q�Wt#|j�|_$t#|j�|_%dS)z�Initializer.

        Args:
            fixer_names: a list of fixers to import
            options: a dict with configuration.
            explicit: a list of fixers to run even if they are explicit.
        NrJrKrIF)�convert�logger)&�fixers�explicit�_default_optionsZcopy�optionsr!r�!python_grammar_no_print_statement�grammarr'�getrK�errors�loggingZ	getLoggerrM�	fixer_log�wroterZDriverrrL�
get_fixers�	pre_order�
post_order�files�bmZ
BottomMatcher�BMZ
bmi_pre_orderZbmi_post_orderrZ
BM_compatibleZ	add_fixerrr,�bmi_pre_order_heads�bmi_post_order_heads)�selfZfixer_namesrQrOr+rrr�__init__�s<





zRefactoringTool.__init__cCs\g}g}�x&|jD�]}t|iidg�}|jdd�d}|j|j�rV|t|j�d�}|jd�}|jdjdd�|D��}yt	||�}Wn$t
k
r�td	||f��YnX||j|j
�}	|	jr�|jd
k	r�||jkr�|jd|�q|jd|�|	jd
k�r|j|	�q|	jdk�r |j|	�qtd|	j��qWtjd�}
|j|
d�|j|
d�||fS)aInspects the options to load the requested patterns and handlers.

        Returns:
          (pre_order, post_order), where pre_order is the list of fixers that
          want a pre-order AST traversal, and post_order is the list that want
          post-order traversal.
        r
r.rNZ_�cSsg|]}|j��qSr)Ztitle)r/r$rrrr2��z.RefactoringTool.get_fixers.<locals>.<listcomp>zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sZpreZpostzIllegal fixer order: %rZ	run_order��key�����)rNr
Zrsplitr�FILE_PREFIX�lenZsplit�CLASS_PREFIX�joinZgetattrZAttributeErrorrGrQrWrO�log_message�	log_debugZorderr�operatorZ
attrgetter�sort)raZpre_order_fixersZpost_order_fixersZfix_mod_pathZmodr0ZpartsZ
class_nameZ	fix_classr+Zkey_funcrrrrY�s8


zRefactoringTool.get_fixerscOs�dS)zCalled when an error occurs.Nr)ra�msg�args�kwdsrrr�	log_errorszRefactoringTool.log_errorcGs|r||}|jj|�dS)zHook to log a message.N)rMZinfo�rarprqrrrrlszRefactoringTool.log_messagecGs|r||}|jj|�dSr5)rMZdebugrtrrrrmszRefactoringTool.log_debugcCsdS)zTCalled with the old version, new version, and filename of a
        refactored file.Nr)ra�old_text�new_text�filename�equalrrr�print_outputszRefactoringTool.print_outputcCs<x6|D].}tjj|�r&|j|||�q|j|||�qWdS)z)Refactor a list of files and directories.N)rrZisdir�refactor_dir�
refactor_file)ra�items�write�
doctests_onlyZdir_or_filerrr�refactor#s
zRefactoringTool.refactorc
Cs�tjd}x�tj|�D]�\}}}|jd|�|j�|j�xH|D]@}|jd�rBtjj|�d|krBtjj||�}	|j	|	||�qBWdd�|D�|dd�<qWdS)z�Descends down a directory and refactor every Python file found.

        Python files are assumed to have a .py extension.

        Files and subdirectories starting with '.' are skipped.
        ZpyzDescending into %sr.rcSsg|]}|jd�s|�qSr-)r)r/Zdnrrrr2>rdz0RefactoringTool.refactor_dir.<locals>.<listcomp>N)
rZextsepZwalkrmrorrZsplitextrkr{)
raZdir_namer}r~Zpy_extZdirpathZdirnamesZ	filenamesrZfullnamerrrrz,s

zRefactoringTool.refactor_dircCs�yt|d�}Wn.tk
r<}z|jd||�dSd}~XnXztj|j�d}Wd|j�Xt|d|d��}t|j	��|fSQRXdS)zG
        Do our best to decode a Python source file correctly.
        ZrbzCan't open %s: %sNrr#��encoding�NN)
�open�OSErrorrsrZdetect_encodingrB�close�_open_with_encodingr<�read)rarw�f�errr�rrr�_read_python_source@s
z#RefactoringTool._read_python_sourcecCs�|j|�\}}|dkrdS|d7}|rn|jd|�|j||�}|jsL||kr`|j|||||�q�|jd|�nH|j||�}|js�|r�|jr�|jt|�dd�|||d�n|jd|�dS)	zRefactors a file.Nr8zRefactoring doctests in %szNo doctest changes in %sr)r}r�zNo changes in %srg)r�rm�refactor_docstringrK�processed_file�refactor_string�was_changed�str)rarwr}r~r;r��output�treerrrr{PszRefactoringTool.refactor_filecCs�t|�}d|krtj|j_zJy|jj|�}Wn4tk
r`}z|jd||jj	|�dSd}~XnXWd|j|j_X||_
|jd|�|j||�|S)aFRefactor a given input string.

        Args:
            data: a string holding the code to be refactored.
            name: a human-readable name for use in error/log messages.

        Returns:
            An AST corresponding to the refactored input stream; None if
            there were errors during the parse.
        rJzCan't parse %s: %s: %sNzRefactoring %s)
rFrrRrrSZparse_stringr"rs�	__class__r�future_featuresrm�
refactor_tree)raZdatarrDr�r�rrrr�gs
zRefactoringTool.refactor_stringcCs�tjj�}|rN|jd�|j|d�}|js2||krB|j|d|�q�|jd�n:|j|d�}|jsj|r~|jr~|jt	|�d|�n
|jd�dS)NzRefactoring doctests in stdinz<stdin>zNo doctest changes in stdinzNo changes in stdin)
�sysZstdinr�rmr�rKr�r�r�r�)rar~r;r�r�rrr�refactor_stdin�s

zRefactoringTool.refactor_stdinc
Cs�x"t|j|j�D]}|j||�qW|j|j|j��|j|j|j��|jj|j	��}�xvt
|j���rАx`|jjD�]R}||ko�||rv||j
tjjdd�|jr�||j
tjjd��x
t||�D�]�}|||kr�||j|�yt|�Wntk
�rw�YnX|j�r(||jk�r(q�|j|�}|r�|j||�}|dk	r�|j|�x,|j�D] }|j�spg|_|jj|��q^W|jj|j	��}x2|D]*}	|	|k�r�g||	<||	j||	��q�Wq�WqvWq\Wx$t|j|j�D]}|j||��q�W|jS)a�Refactors a parse tree (modifying the tree in place).

        For compatible patterns the bottom matcher module is
        used. Otherwise the tree is traversed node-to-node for
        matches.

        Args:
            tree: a pytree.Node instance representing the root of the tree
                  to be refactored.
            name: a human-readable name for this tree.

        Returns:
            True if the tree was modified, False otherwise.
        T)rfZreversereN)rrZr[Z
start_tree�traverse_byr_r`r^ZrunZleavesZanyr(rNrorZBaseZdepthZkeep_line_orderZ
get_linenor&ZremoverZ
ValueErrorZfixers_applied�match�	transformr9rr*Zfinish_treer�)
rar�rr+Z	match_set�node�results�newZnew_matchesZfxrrrrr��sJ



$zRefactoringTool.refactor_treecCs^|sdSxP|D]H}xB||jD]4}|j|�}|r|j||�}|dk	r|j|�|}qWqWdS)aTraverse an AST, applying a set of fixers to each node.

        This is a helper method for refactor_tree().

        Args:
            fixers: a list of fixer instances.
            traversal: a generator that yields AST nodes.

        Returns:
            None
        N)rr�r�r9)rarNZ	traversalr�r+r�r�rrrr��s


zRefactoringTool.traverse_bycCs�|jj|�|dkr.|j|�d}|dkr.dS||k}|j||||�|r`|jd|�|js`dS|rv|j||||�n|jd|�dS)zR
        Called when a file has been refactored and there may be changes.
        NrzNo changes to %szNot writing changes to %s)r\rr�ryrmrK�
write_file)rarvrwrur}r�rxrrrr��szRefactoringTool.processed_filec%Cs�yt|d|d�}Wn.tk
r@}z|jd||�dSd}~XnXzHy|jt|��Wn0tk
r�}z|jd||�WYdd}~XnXWd|j�X|jd|�d|_dS)z�Writes a string to a file.

        It first shows a unified diff between the old text and the new text, and
        then rewrites the file; the latter is only done if the write option is
        set.
        Zwr�zCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)r�r�rsr}r=r�rmrX)rarvrwrur�r�r�rrrr�
s$
zRefactoringTool.write_filez>>> z... c
Csg}d}d}d}d}x�|jdd�D]�}|d7}|j�j|j�r�|dk	r\|j|j||||��|}|g}|j|j�}	|d|	�}q"|dk	r�|j||j�s�|||jj�dkr�|j	|�q"|dk	r�|j|j||||��d}d}|j	|�q"W|dk	�r|j|j||||��dj
|�S)a�Refactors a docstring, looking for doctests.

        This returns a modified version of the input string.  It looks
        for doctests, which start with a ">>>" prompt, and may be
        continued with "..." prompts, as long as the "..." is indented
        the same as the ">>>".

        (Unfortunately we can't use the doctest module's parser,
        since, like most parsers, it is not geared towards preserving
        the original source.)
        NrT�Zkeependsrr8rc)�
splitlinesZlstripr�PS1r*�refactor_doctestZfind�PS2�rstriprrk)
rar;rwZresult�blockZblock_lineno�indent�lineno�line�irrrr�%s:



z"RefactoringTool.refactor_docstringc
sy�j||��}Wndtk
rv}zH�jjtj�rRx|D]}�jd|jd��q8W�jd|||j	j
|�|Sd}~XnX�j||��r
t|�j
dd�}|d|d�||dd�}	}|djd�s�|dd7<��j|jd�g}|�r
|��fd	d
�|D�7}|S)
z�Refactors one doctest.

        A doctest is given as a block of lines, the first of which starts
        with ">>>" (possibly indented), while the remaining lines start
        with "..." (identically indented).

        z
Source: %sr8z+Can't parse docstring in %s line %s: %s: %sNTr�rrcsg|]}��j|�qSr)r�)r/r��r�rarrr2jrdz4RefactoringTool.refactor_doctest.<locals>.<listcomp>rgrg)�parse_blockr"rMZisEnabledForrVZDEBUGrmr�rsr�rr�r�r�rr�Zpop)
rar�r�r�rwr�r�r�r�Zclippedrr�rr�Ps$
"z RefactoringTool.refactor_doctestcCs�|jrd}nd}|js$|jd|�n&|jd|�x|jD]}|j|�q8W|jrt|jd�x|jD]}|j|�qbW|jr�t|j�dkr�|jd�n|jdt|j��x&|jD]\}}}|j|f|�|�q�WdS)	N�werez
need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:rzThere was 1 error:zThere were %d errors:)rXr\rlrWrUri)rar�ZfileZmessagerprqrrrrr�	summarizems$
zRefactoringTool.summarizecCs"|jj|j|||��}t�|_|S)z�Parses a block into a tree.

        This is necessary to get correct line number / offset information
        in the parser diagnostics and embedded into the parse tree.
        )rZparse_tokens�	wrap_toksrCr�)rar�r�r�r�rrrr��szRefactoringTool.parse_blockccshtj|j||�j�}xN|D]F\}}\}}\}	}
}||d7}|	|d7}	||||f|	|
f|fVqWdS)z;Wraps a tokenize stream to systematically modify start/end.rN)rr@�	gen_linesZ__next__)rar�r�r�r)rrEZline0Zcol0Zline1Zcol1Z	line_textrrrr��s
zRefactoringTool.wrap_toksccs�||j}||j}|}xV|D]N}|j|�r@|t|�d�Vn(||j�dkrXdVntd||f��|}qWx
dVqrWdS)z�Generates lines as expected by tokenize from a list of lines.

        This strips the first len(indent + self.PS1) characters off each line.
        Nr8zline=%r, prefix=%rrc)r�r�rrir�ZAssertionError)rar�r�Zprefix1Zprefix2Zprefixr�rrrr��s



zRefactoringTool.gen_linesr��FFr�r�)F)NFNr5)rrrrPrjrhrbrYrsrlrmryrrzr�r{r�r�r�r�r�r�r�r�r�r�r�r�r�r�rrrrrI�s:
4(
	


O

+
rIc@r)�MultiprocessingUnsupportedNrrrrrr��rr�csBeZdZ�fdd�Zd�fdd�	Z�fdd�Z�fd	d
�Z�ZS)�MultiprocessRefactoringToolcs"tt|�j||�d|_d|_dSr5)�superr�rb�queue�output_lock�rarq�kwargs�r�rrrb�sz$MultiprocessRefactoringTool.__init__Frcs|dkrtt��j|||�Syddl�Wntk
r@t�YnX�jdk	rTtd���j��_�j	��_
��fdd�t|�D�}z.x|D]}|j�q�Wtt��j|||�Wd�jj
�xt|�D]}�jjd�q�Wx|D]}|j�r�|j
�q�Wd�_XdS)Nrrz already doing multiple processescsg|]}�j�jd��qS))Ztarget)ZProcess�_child)r/r���multiprocessingrarrr2�r3z8MultiprocessRefactoringTool.refactor.<locals>.<listcomp>)r�r�rr�ZImportErrorr�r�ZRuntimeErrorZ
JoinableQueueZLockr�ZrangeZstartrk�putZis_alive)rar|r}r~Z
num_processesZ	processesr$r�r�r�rr�s2









z$MultiprocessRefactoringTool.refactorc
sR|jj�}xB|dk	rL|\}}ztt|�j||�Wd|jj�X|jj�}qWdSr5)r�rTr�r�r{Z	task_done)raZtaskrqr�r�rrr��s


z"MultiprocessRefactoringTool._childcs2|jdk	r|jj||f�ntt|�j||�SdSr5)r�r�r�r�r{r�r�rrr{�s

z)MultiprocessRefactoringTool.refactor_file)FFr)rrrrbrr�r{Z
__classcell__rrr�rr��s
r�)T)rr)(rHZ
__author__rr�rVrnr%rAZ	itertoolsrZpgen2rrrZ
fixer_utilrrcrrr	r]rr"rrr,r4r6Zversion_infoZcodecsr�r�r<r=rFrGZobjectrIr�r�rrrrZ<module>	sF



(