Try to use better robots.txt file.I have mean the same as you but the search engines add your site but if you don not use korekt the meta tags and robots it will been on the 7-8 page. Do you now what I mean ?
https://www.woltlab.com/de/ It is on german and it cost 25 Â you can download a lite package from this hotlink https://www.woltlab.com/de/ If you will to try WoltLab you can try it here User Demo Admin Demo Have fun and post here about this board