diff options
| author | Rodrigo Arias Mallo <rodarima@gmail.com> | 2024-01-01 23:40:52 +0100 |
|---|---|---|
| committer | Rodrigo Arias Mallo <rodarima@gmail.com> | 2024-01-01 23:40:52 +0100 |
| commit | 5ea943a5e789222472e45864e119cf786498bfcd (patch) | |
| tree | ea307589de0fdb202474ad4d07c0bef7fe1c53e8 /old/robots.txt | |
Import original dillo.org website into old/
Diffstat (limited to 'old/robots.txt')
| -rw-r--r-- | old/robots.txt | 19 |
1 files changed, 19 insertions, 0 deletions
diff --git a/old/robots.txt b/old/robots.txt new file mode 100644 index 0000000..151b427 --- /dev/null +++ b/old/robots.txt @@ -0,0 +1,19 @@ +User-agent: AhrefsBot +Disallow: / +User-agent: dotbot +Disallow: / +User-agent: SiteExplorer +Disallow: / +User-agent: BLEXBot +Disallow: / +User-agent: spbot +Disallow: / +User-agent: MJ12bot +Disallow: / +User-agent: SemrushBot +Disallow: / +User-agent: SemrushBot-SA +Disallow: / +User-agent: * +Disallow: /download/ +Disallow: /test/mysql-6.0.doc.tar.bz2 |