�
��[ig � �v � d Z ddlZddlZddlmZ ddlmZ e� Zdedefd�Z ded ede
fd
�Z ded ede
fd �Z y) z+F1 evaluation metrics (ported from LoCoMo).� N)�Counter)�
PorterStemmer�s�returnc � � t | � j dd� } t j dd| t j �� } dj d� | D � � } dj | j
� j � � } | S )z Normalize answer for comparison.�,� z\b(a|an|the|and)\b� )�flagsc 3 �F K � | ] }|t j vs�|�� � y �w�N)�string� punctuation)�.0�chs �0/home/spoj/adaptive_memory/bench/eval_metrics.py� <genexpr>z#normalize_answer.<locals>.<genexpr> s � �� �?�Q�r�"�F�,>�,>�">��Q�s �!�!)�str�replace�re�sub�
IGNORECASE�join�lower�split)r s r �normalize_answerr sc � � �A����s�B� �A�
���$�c�1�B�M�M�B�A�
���?�Q�?�?�A� ������� � �"�#�A� �H� �
prediction� ground_truthc �� � t | � j � D �cg c] }t j |� �� }}t |� j � D �cg c] }t j |� �� }}|r|syt |� t |� z }t |j
� � }|dk( ry|t |� z }|t |� z }d|z |z ||z z S c c}w c c}w )z$Token-level F1 with Porter stemming.� r � )r r �ps�stemr �sum�values�len) r r �w� pred_tokens� gt_tokens�common�num_same� precision�recalls r �f1_scorer/ s� � �'7�
�'C�'I�'I�'K�L�'K�!�2�7�7�1�:�'K�K�L�%5�l�%C�%I�%I�%K�L�%K�������%K�I�L� �i��
�[�
!�G�I�$6�
6�F��6�=�=�?�#�H��1�}���3�{�+�+�I�
��I��
&�F�
� �M�F� "�y�6�'9� :�:�� M��Ls � C� C c �` �� | j d� D �cg c] }|j � �� }}|j d� D �cg c] }|j � �� }}|syg }|D ]+ �|rt �fd�|D � � nd}|j |� �- t |� t |� z S c c}w c c}w )z0F1 for multi-answer questions (comma-separated).r r! c 3 �6 �K � | ] }t |�� �� � y �wr
)r/ )r �pred�gts �r r zf1_multi.<locals>.<genexpr>5 s �� �� �>�+�$�8�D�"�%�+�s �)r �strip�max�appendr% r' ) r r �p� predictions�g�
ground_truths�scores�bestr3 s @r �f1_multir= + s� �� �&0�&6�&6�s�&;�<�&;��1�7�7�9�&;�K�<�(4�(:�(:�3�(?�@�(?�1�Q�W�W�Y�(?�M�@� ��
�F���BM�s�>�+�>�>�SV���
�
�d�� � �v�;��V�� $�$�� =��@s �B&�B+)
�__doc__r r � collectionsr � nltk.stemr r# r r �floatr/ r= � r r �<module>rC sb �� 1� �
� � #��_��
��
��
�;�� ;�C� ;�E� ;�&
%��
%�C�
%�E�
%r