Desarrollo y diseño

Google no encuentra robots.txt

Thanat0s
Thanat0s #1 Sep '13 Cofrade

Hola gente, estoy teniendo un problema bastante curioso con el archivo robots.txt

Resulta que yo puedo acceder al archivo sin problemas y al hacer un "Fetch as Google" desde GWT, funciona perfectamente.

Pero cuando google lo intenta hacer de manera automática, pasa lo siguiente:

  • DNS: ok
  • Server Connectivity: ok
  • Robots.txt Fetch: Unreachable

Los logs del servidor apuntan a que todo ha funcionado bien:
66.249.66.161 - - [01/Sep/2013:05:05:01 +0200] "GET /robots.txt HTTP/1.1" 200 57
66.249.66.161 - - [01/Sep/2013:12:08:52 +0200] "GET /robots.txt HTTP/1.1" 200 57
64.246.165.50 - - [02/Sep/2013:01:38:22 +0200] "GET /robots.txt HTTP/1.0" 200 57
66.249.66.161 - - [03/Sep/2013:17:14:21 +0200] "GET /robots.txt HTTP/1.1" 200 57
66.249.66.161 - - [04/Sep/2013:05:07:55 +0200] "GET /robots.txt HTTP/1.1" 200 57
207.241.226.239 - - [04/Sep/2013:13:19:13 +0200] "GET /robots.txt HTTP/1.1" 200 57
54.227.169.60 - - [04/Sep/2013:15:27:38 +0200] "HEAD /robots.txt HTTP/1.1" 200 -
54.227.169.60 - - [04/Sep/2013:15:27:38 +0200] "HEAD /robots.txt HTTP/1.1" 200 -
54.227.169.60 - - [04/Sep/2013:15:27:38 +0200] "HEAD /robots.txt HTTP/1.1" 200 -
184.72.182.251 - - [04/Sep/2013:15:27:39 +0200] "HEAD /robots.txt HTTP/1.1" 200 -
184.72.182.251 - - [04/Sep/2013:15:27:39 +0200] "HEAD /robots.txt HTTP/1.1" 200 -
184.72.182.251 - - [04/Sep/2013:15:27:40 +0200] "HEAD /robots.txt HTTP/1.1" 200 -
54.227.169.60 - - [04/Sep/2013:15:27:40 +0200] "GET /robots.txt HTTP/1.1" 200 57
184.72.182.251 - - [04/Sep/2013:15:27:41 +0200] "GET /robots.txt HTTP/1.1" 200 57
204.236.235.245 - - [04/Sep/2013:19:42:39 +0200] "GET /robots.txt HTTP/1.0" 200 57
66.249.74.57 - - [05/Sep/2013:03:17:16 +0200] "GET /robots.txt HTTP/1.1" 200 57
66.249.75.161 - - [06/Sep/2013:17:14:20 +0200] "GET /robots.txt HTTP/1.1" 200 57
180.76.5.8 - - [07/Sep/2013:08:32:50 +0200] "GET /robots.txt HTTP/1.1" 200 57
66.249.78.10 - - [08/Sep/2013:02:52:14 +0200] "GET /robots.txt HTTP/1.1" 200 57
66.249.66.161 - - [09/Sep/2013:17:14:18 +0200] "GET /robots.txt HTTP/1.1" 200 57

66.249.66.161 - - [10/Sep/2013:17:41:08 +0200] "GET /robots.txt HTTP/1.1" 200 61
199.16.156.125 - - [11/Sep/2013:02:13:35 +0200] "GET /robots.txt HTTP/1.1" 200 61
199.16.156.125 - - [11/Sep/2013:02:13:35 +0200] "GET /robots.txt HTTP/1.1" 200 61
54.221.107.65 - - [11/Sep/2013:02:13:38 +0200] "GET /robots.txt HTTP/1.1" 200 61
180.76.5.147 - - [11/Sep/2013:04:43:48 +0200] "GET /robots.txt HTTP/1.1" 200 61
66.249.66.161 - - [11/Sep/2013:14:08:40 +0200] "GET /robots.txt HTTP/1.1" 200 61
66.249.66.161 - - [12/Sep/2013:17:14:17 +0200] "GET /robots.txt HTTP/1.1" 200 61
2.139.173.154 - - [12/Sep/2013:17:46:50 +0200] "GET /robots.txt HTTP/1.1" 200 61
2.139.173.154 - - [12/Sep/2013:17:46:54 +0200] "GET /robots.txt HTTP/1.1" 200 61
2.139.173.154 - - [12/Sep/2013:17:46:57 +0200] "GET /robots.txt HTTP/1.1" 200 61
66.249.66.161 - - [12/Sep/2013:18:02:28 +0200] "GET /robots.txt HTTP/1.1" 200 61
66.249.66.161 - - [12/Sep/2013:18:03:16 +0200] "GET /robots.txt HTTP/1.1" 200 61
2.139.173.154 - - [13/Sep/2013:09:23:21 +0200] "GET /robots.txt HTTP/1.1" 304 -
188.40.139.10 - - [13/Sep/2013:09:23:31 +0200] "GET /robots.txt HTTP/1.1" 200 61
208.113.162.84 - - [13/Sep/2013:09:40:52 +0200] "GET /robots.txt HTTP/1.0" 200 61
208.113.162.84 - - [13/Sep/2013:09:45:30 +0200] "GET /robots.txt HTTP/1.0" 200 68
2.139.173.154 - - [13/Sep/2013:09:47:26 +0200] "GET /robots.txt HTTP/1.1" 200 68
188.40.139.10 - - [13/Sep/2013:09:47:37 +0200] "GET /robots.txt HTTP/1.1" 200 68
66.249.81.10 - - [13/Sep/2013:09:49:48 +0200] "GET /robots.txt HTTP/1.1" 200 68
2.139.173.154 - - [13/Sep/2013:09:49:51 +0200] "GET /robots.txt HTTP/1.1" 304 -
66.249.66.161 - - [13/Sep/2013:09:50:22 +0200] "GET /robots.txt HTTP/1.1" 200 68
2.139.173.154 - - [13/Sep/2013:13:17:31 +0200] "GET /robots.txt HTTP/1.1" 200 68
2.139.173.154 - - [13/Sep/2013:13:31:16 +0200] "GET /robots.txt HTTP/1.1" 304 -
2.139.173.154 - - [13/Sep/2013:13:40:00 +0200] "GET /robots.txt HTTP/1.1" 200 68
66.249.66.161 - - [13/Sep/2013:13:41:36 +0200] "GET /robots.txt HTTP/1.1" 200 68
2.139.173.154 - - [13/Sep/2013:13:44:03 +0200] "GET /robots.txt HTTP/1.1" 200 68

No sé qué puede haber mal, ¿alguien tiene alguna sugerencia?

Gracias.

Thanat0s
Thanat0s #2 Sep '13 Cofrade

Por si a alguien le interesa:
https://productforums.google.com/d/msg/webmasters/oRIWr052JTE/37aYtirGNLYJ

13500
13500 #3 Sep '13

"Error was missing http:// in the sitemap url."

al final era eso?

1 respuesta
Thanat0s
Thanat0s #4 Sep '13 Cofrade

#3 No, sigue fallando.

1 respuesta
13500
13500 #5 Sep '13

#4 puedes pegar el texto de tu robots.txt o es confidencial?

1 respuesta
Thanat0s
Thanat0s #6 Sep '13 Cofrade

#5 Es casi default, no hay ningún problema.

User-agent: *
Disallow:
Sitemap: http://www.videoacta.es/sitemap.xml

1 respuesta
13500
13500 #7 Sep '13

#6 quita ese Disallow: nada y pon

allow: /

1 1 respuesta
Thanat0s
Thanat0s #8 Sep '13 Cofrade

#7 Ese cambio no le gusta mucho al test de GWT.

Test results
Url
http://www.videoacta.es/

Googlebot
Allowed by line 2: Allow: /
Detected as a directory; specific files may have different restrictions

Googlebot-Mobile
Allowed by line 2: Allow: /
Detected as a directory; specific files may have different restrictions

robots.txt analysis

Edit: calla, no lo había leído detenidamente, si que le gusta el cambio. Voy a dejarlo así a ver.

1 1 respuesta
elkaoD
elkaoD #9 Sep '13

#8 apuesto a que es eso xD Típica chorrada que pasa desapercibida.

Thanat0s
Thanat0s #10 Sep '13 Cofrade

Bueno, algo ha mejorado, ahora ya no da el unreachable, ahora simplemente está inaccesible xD

A ver si actualiza la gráfica de datos del fetch.

Usuarios habituales

  • Thanat0s
  • elkaoD
  • 13500

Tags