API Reference
Class
RobotsTxtAnalysis
Downloads and parses robots.txt files.
Inheritance
- Object
- RobotsTxtAnalysis
Usage
This type appears in these public API surfaces even when no hand-authored example is attached directly to the page.
Returned or exposed by
- Property DomainHealthCheck.RobotsTxtAnalysis
Accepted by parameters
- Method RobotsTxtNarrative.Build
Constructors
public RobotsTxtAnalysis() #Methods
public async Task AnalyzeAsync(String domain, InternalLogger logger = null, CancellationToken cancellationToken = null) #Returns:
TaskRetrieves and parses robots.txt for domain.
Parameters
- domain System.String
- logger DomainDetective.InternalLogger = null
- cancellationToken System.Threading.CancellationToken = null
Inherited Methods
public override Boolean Equals(Object obj) #Returns:
BooleanInherited from Object
Parameters
- obj Object
Properties
public TimeSpan CacheDuration { get; set; } #Maximum time cached results are kept.
public String Domain { get; set; } #Domain that was analyzed.
public Boolean RecordPresent { get; set; } #True when a robots.txt file was found.
public Boolean FallbackUsed { get; set; } #Indicates a fallback to HTTP was used.
public String Url { get; set; } #URL the robots.txt was downloaded from.
public RobotsFile Robots { get; set; } #Parsed robots.txt data.
public Dictionary<KnownAiBot, List<String>> AiBots { get; } #List of AI user-agents found in the file grouped by bot type.
public Boolean HasAiBotRules { get; } #Indicates whether AI bot directives are present.
public virtual List<Assessment> Assessments { get; } #Gets the assessments value.