Added user agant in robots.txt

Esi Zaimirequested to merge
#356 into master

Added user agent disallow function.

Merge request reports