robots_txt | Lightweight robots.txt parser and generator written in Rust | Sitemap library

Seemingly conflicting descriptions given in authoritative documentation sources.

A Standard for Robot Exclusion:

('record' refers to each user-agent block)

"The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains lines of the form ...".

Google's Robot.txt Specifications:

"... Note the optional use of white-space and empty lines to improve readability."

So -- based on documentation that we have available to us -- is this empty line here mandatory?

I'm trying to start working with a PHP webapp (written by someone else) on a local Apache server. Every time I try to open any .php file in the project folder, I get error 404.

I checked that the server is working by opening other php files in other projects, no problems there. After researching possible causes, I suspect that it might be the .htaccess file but I'm not sure what is the specific problem. The code in this file is the following:

For example google's page shows a description:
In their case the text is

Search the world's information, including webpages, images, video's and more. Google has many special features to help you find exactly what you're looking ...

I've looked at the source of their page to find how this text is determined but could find nothing. Google also has a nice page explaining how to make descriptions, but never specify where to put the description.

Someone told me the description should be in the robots.txt, but when looking at the specification of robots.txt it only has four keywords:
- user-agent (start of group)
- disallow (only valid as a group-member record)
- allow (only valid as a group-member record)
- sitemap (non-group record)

None of them are description or search tags.

The Google Robots.txt Specification states that a robots txt URL http://example.com/robots.txt is not valid for domain https://example.com. Presumably the reverse it also true.

It also has this to say about following redirects when requesting a robots.txt:

3xx (redirection)

Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404. Handling of robots.txt redirects to disallowed URLs is undefined and discouraged. Handling of logical redirects for the robots.txt file based on HTML content that returns 2xx (frames, JavaScript, or meta refresh-type redirects) is undefined and discouraged.

Say I set up a website so that all requests on http are redirected permanently to equivalent on https. Google will request http://example.com/robots.txt and follow the redirect to https://example.com/robots.txt. Will this file be the valid robots.txt for the http site, because that was the original request, or will Google think there is no valid robots.txt for the http site?

I have a server running Apache 2.2.24 and CodeIgniter 2.1.0 .

I want to handle requests for /robots.txt dynamically, using CodeIgniter, but the only solution I have found so far is to use .htaccess to issue a temporary redirect that tells clients to fetch /robots_txt instead (note the underscore).