ModifiedOn will always be the user that marked a task as completed. When a user marks a task as completed, the form becomes read-only. Adding (relating) a notes to an entity doesn't change the modified-on field of the related entity.
Hope this helps!
If my response helps you in finding your answer then please click 'Mark as Answer' and 'Vote as Helpful'