Meta's new ad terms let it crawl my website. Can I block it without breaking my ads?

TofuAnswer by the Tofu team · checked

Mostly, yes. Meta's updated Commercial Terms and Self-Serve Ad Terms let it crawl sites "associated with your products or services" to develop and maintain "advertising and business products." They don't mention robots.txt, and Meta's crawler page says to block its crawlers there. Meta doesn't say that blocking meta-externalagent (AI training and indexing) or meta-externalads (ads and business products) affects ad delivery. Keep facebookexternalhit allowed: it builds link previews and may skip robots.txt for security checks anyway. The US and UK English banners both say the update takes effect November 20, 2026.

As of October 2026.

What changed

BeforeNow (updated terms, checked October 27, 2026)
Commercial TermsNo crawling clause in the current version, effective March 6, 2024A clause authorizing "automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents)" and "test credentials" to crawl your sites
Self-Serve Ad TermsThe current version (effective August 22, 2025) already lets Meta crawl your "domains, webpages or URLs" for "developing and improving advertising and other business-related products and related uses, research and training"The same clause as the Commercial Terms, word for word
What Meta may do with the contentEarlier preview: "Meta may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products," as quoted by MediaNama, and by PPC Land citing The Verge"in order to provide, develop, improve, research, and maintain advertising and business products" and "maintain the integrity and security of Meta Company Products." The word "train" is gone
robots.txtNot mentionedStill not mentioned
Effective dateNot applicableUS English banners and footers: November 20, 2026. UK English banners: 20 November 2026. UK English footers: "Effective date: 30 October 2026"

Here is the full clause as it reads on Meta's US English pages. It sits in the "Licenses & Rights You Provide Meta" section of the Commercial Terms, and the updated Self-Serve Ad Terms repeat it:

You authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs (and any content available through them) that are associated with your products or services, or that are made available through your or users' use of Meta Company Products and services, and to periodically pull updates from such domains, web pages, or URLs in order to provide, develop, improve, research, and maintain advertising and business products (including business, commercial and related products to help you create and deliver ads and other commercial content, provide users information about your products and services, and connect you with users who may be interested in your products and services) and maintain the integrity and security of Meta Company Products.

The UK English pages use the same words with British spelling ("authorise," "analyse") and fewer commas.

How the wording changed. PPC Land reported the clause on October 16, 2026. The earlier preview had a second sentence letting Meta use the content to "train" its products. PPC Land says The Verge reported its removal on October 17, yet PPC Land still found the old wording on the UK English preview on October 18. When we checked on October 27, the US and UK English previews both had the new wording. Set against PPC Land's quote of the draft, the crawl permission itself (bots, test credentials, periodic updates) is unchanged.

A Meta spokesperson said, as quoted by MediaNama: "As always, content owners can use robots.txt to signal their preferences. This is not about scraping news sites, and these updates don't give us new rights that supersede those preferences." That statement is not in the terms.

Meta's Ad Creative Generative AI Terms (effective May 6, 2024) already let it crawl URLs you provide "for your use in connection with your ads." For the AI-features half of this update and why the date moved, see turning off Meta's AI edits before the new terms.

Who it affects

  • Every business using Meta for business purposes. The Commercial Terms cover "using ads or marketing messages," "managing a Page" and "using any business product, such as Ads Manager or Meta Business Suite." You don't have to run ads for them to apply.
  • More than your landing pages. The clause covers "domains" as well as pages, and URLs "made available through your or users' use of Meta Company Products," not only the links in your ads.
  • Test credentials. The clause also lets Meta "create test credentials" to access those sites. The terms don't define them.
  • No partial opt-out. The terms say continuing to use Meta's products for business after notice of an update means "you agree to be bound by them," and "If you do not agree to the updated terms, you may stop all access or use of our Products for business or commercial purposes."

This page describes what the terms and Meta's crawler page say. It isn't legal advice.

What to do now

Meta's web crawlers page lists five crawlers. Here is what each does and what Meta says about blocking it:

Crawler (user agent)What Meta says it doesrobots.txtWhat blocking it means for your ads
facebookexternalhitFetches links shared on Facebook, Instagram or Messenger for their title, description and thumbnail"might bypass robots.txt when performing security or integrity checks, such as checking for malware"Risks blank link previews. Meta says that unless its crawler can crawl your content within a few seconds, "Facebook will be unable to display the content"
meta-externalads"crawls the web for use cases such as improving advertising and other business-related products and services"One of the two crawlers in Meta's disallow exampleMeta doesn't say
meta-externalagent"crawls the web for use cases such as training foundation AI models or improving products by indexing content directly"One of the two crawlers in Meta's disallow exampleMeta doesn't describe it as an ads crawler
meta-webindexerImproves Meta AI search results; allowing it "helps us cite and link to your content in Meta AI's responses"Meta lists no bypassMeta describes it for Meta AI search, not ads
meta-externalfetcherFetches individual links "at a user's request," including for AI agents completing tasks"may bypass robots.txt rules"Meta describes it for user requests, not ads
  1. Keep facebookexternalhit allowed. It builds link previews. Meta's requirements: allowlist its user agents or IP addresses, serve gzip and deflate encodings, put Open Graph tags in the first 1 MB, and respond within a few seconds.
  2. Block meta-externalagent if you don't want your site used to train Meta's AI models. Add a disallow for it in robots.txt, as in the example below. Meta says the same crawler also indexes content "directly" to improve products, so a block covers both uses.
  3. Decide on meta-externalads with care. It is the crawler named for advertising, and Meta doesn't document what blocking it does to review, delivery or features. Some Advantage+ creative enhancements, such as Reveal details over time and Show spotlights, show details, text or images from your website in your ads; Meta doesn't say which crawler they use. If you block it, watch your ads for missing details and new rejections.
  4. Keep landing pages reachable. Meta's ad review can include the "ad destination," and its policy checklist says the site an ad links to should be "functioning properly" and "clearly accessible and relevant to your ad." If you create ads through the Marketing API with image URLs, Meta's error list includes error 100, subcode 3858258, for an image that "couldn't be downloaded" because it may be "blocked by a robots.txt file."
  5. Write the rules and give them a day. Meta's example gives each crawler its own User-agent line followed by Allow and Disallow rules. A file that blocks meta-externalagent from the whole site and keeps meta-externalads out of one folder looks like this: `` User-agent: meta-externalagent Disallow: / User-agent: meta-externalads Disallow: /private/ `` Meta says to "allow up to 24 hours" for changes, because its crawlers may cache robots.txt for that long. Then run a page through the Sharing Debugger to check that link previews still work.
  6. Know what robots.txt can't do. Meta's crawler page presents it as the way site owners "indicate their preferences," but the terms say nothing about it, and two of Meta's crawlers may bypass it. For questions, Meta's crawler page gives [email protected].

Dates to watch

DateWhat happens
March 6, 2024Effective date of the current Commercial Terms, which have no crawling clause
August 22, 2025Effective date of the current Self-Serve Ad Terms, whose crawling clause includes "research and training"
October 17, 2026The Verge reports the "train" sentence removed (per PPC Land)
October 27, 2026Our check: the US and UK English previews both carry the new wording
October 30, 2026The date still printed in the UK English documents' footers
November 20, 2026The date in the US and UK English banners and the US English footers
Sources (16)

Ask Tofu next

Pick a question and Tofu Ads Agent answers it in chat.

Related answers

More on