Preventing Woke AI in the Federal Government
www.whitehouse.gov
By the authority vested in me as President by the Constitution and the laws of the United States of America, it is hereby ordered: Section 1.  Purpose.
  • @[email protected]English
    4•2 days ago

    If your reverse proxy is Traefik I would suggest This plugin which pulls the robot.txt from This GitHub repository.

    I honestly should’ve setup a robots.txt a long time ago.

    • @[email protected]English
      5•2 days ago

      Unfortunately robots.txt only stops the well behaved scrapers. Even with disallow all, you’ll still get loads of bots. Setting up the web server to block those user agents would work a bit better, but even then there’s bots out there crawling using regular browser user agents.