# About CopyleaksBot

> Information for webmasters, SEOs, and developers about CopyleaksBot, the web crawler used by Copyleaks.

CopyleaksBot is the official web crawler for Copyleaks. Its purpose is to discover and crawl publicly available web pages for our search services. We are committed to respecting the rules set forth by webmasters in their robots.txt files.

## User Agent

CopyleaksBot identifies itself with the following User-Agent string in its HTTP requests:

```
CopyleaksBot/1.0
```

## How to Control CopyleaksBot

To limit which pages Copyleaks can index, use your website's robots.txt file. CopyleaksBot fully respects the robots.txt standard. You can use it to prevent our bot from indexing your entire site, specific directories, or individual pages. The User-agent token for our bot is `CopyleaksBot`.

<Steps>
  <Step title="Block CopyleaksBot from your entire site">
    Add the following to your robots.txt file:

    ```txt
    User-agent: CopyleaksBot
    Disallow: /
    ```
  </Step>

  <Step title="Block CopyleaksBot from a specific directory">
    Add the following to your robots.txt file:

    ```txt
    User-agent: CopyleaksBot
    Disallow: /private-directory/ 
    ```
  </Step>

  <Step title="Block CopyleaksBot from a single page">
    Add the following to your robots.txt file:

    ```txt
    User-agent: CopyleaksBot
    Disallow: /path/to/page.html
    ```
  </Step>
</Steps>

## Related Resources

<CardGroup cols={2}>
  <Card title="Contact Support" icon="robot" href="https://help.copyleaks.com/hc/en-us/requests/new">Get help with CopyleaksBot or other questions from our support team.</Card>
</CardGroup>