WP Content Crawler logo
WP Content Crawler

Get content from almost any site, automatically!

WP Content Crawler is a WordPress plugin that helps you get content from other websites, into your website.

Save anything.

How it works

It is all about CSS selectors

1
Websites contain many sections
2
Sections are defined with HTML elements
3
HTML elements have classes and attributes
4
CSS selectors are used to select specific elements by using their classes and attributes
5
You specify which element to use via CSS selectors
6
CSS selectors can be found with Visual Inspector by clicking to elements
7
After settings are configured, the plugin collects URLs and saves posts automatically via WP-Cron
Websites contain many sections

Visual Inspector

Find CSS selectors of elements by clicking to them

Find unique elements

Find unique elements

Unique CSS selectors select only one element. This is useful, for example, for finding titles and featured images.

Find similar elements

Find similar elements

The "similar" mode finds a CSS selector that selects the items similar to the clicked item. This is useful, for example, for finding post tags and product attributes.

Find containing elements

Find containing elements

The "contains" mode finds a CSS selector that selects the element containing all the clicked elements. This is useful, for example, for finding post content and excerpt.

See the visual inspector in action
This video demonstrates the selection modes of the visual inspector.

Automatic

Configure the settings and let the plugin handle the rest

Manipulations

Prepare the data in the way you want

Find and replace

Find and replace

Find and replace anything in the target page. You can also use regular expressions. You can use dedicated options to find and replace in the post title, content, tags, post meta, and others…

Learn more about finding and replacing
Advanced text manipulations

Advanced text manipulations

Make text upper case, lower case, title case, camel case, snake case, kebab case, and more… You can also limit the length by character count or word count.

Learn more about advanced text manipulations

Test everything

Easily test your configurations to make sure everything works as expected

Quick tests

Quick tests

Use the magnifier button to quickly test the configurations. You can test find-replace options, CSS selectors, formulas, filter commands, and more… You can also enable caching for the quick tests to make them even faster.

Warnings

Warnings

When a configuration is wrong, you get a warning that tells you why it is wrong. By this way, you can fix it right then and there.

Test the settings altogether

Test the settings altogether

The Site Tester page helps you check if your configurations work as expected. You just enter a category or post page URL and hit the test button. The plugin crawls the page and shows you all the details it finds in the page, along with some performance information and, if any, warnings.

Learn more about testing the settings
See the Site Tester in action
This video demonstrates the features of the Site Tester page.

Translate and spin

Automatically translate and spin specific parts of the posts

Spin

Spin

Spinning is like paraphrasing, keeping the same meaning while using different words. You can automatically spin the posts by using one of the following paid services. You can also choose which parts of the post should be spun.

Learn more about spinning

Do things conditionally!

Perform certain operations if a certain condition is met

A filter that assigns the status of post as "draft" if the title has less than 20 characters

The filter above assigns the status of the post as "draft" if the crawled post's title has less than 20 characters. Using many condition commands available in the plugin, you can create complex conditions. You can combine the conditions with "and" and "or" operators as well. When the condition is met, you can do multiple things, such as sending an email notification to yourself, adding the post to a specific category, finding and replacing things, assigning a specific featured image, calculating a value, and many more.

Learn how to use the filters
This video tutorial explains the filters in detail, with examples

Save WooCommerce products

Use the options dedicated to WooCommerce products to easily save the product details

Dedicated options

Dedicated options

The WooCommerce section's design is similar to the design of the original WooCommerce options. Here, instead of entering the values, you define CSS selectors. You can create simple and external products. You can also mark the product as virtual and/or downloadable. You can retrieve the downloadable file from the target page as well.

Learn more about WooCommerce options
Dedicated test results

Dedicated test results

When you test a product page via the Site Tester, the test results contain a dedicated WooCommerce section where you can see all the product details retrieved from the target page, so that you can ensure that the products will be created as expected.

Crawling WooCommerce products
This video tutorial walks you through configuring the site settings to crawl products from eBay

Track what's going on

The dashboard page helps you understand what's going on in the background

Track active sites

Track active sites

See the active sites, when they were crawled the latest, how many posts were crawled, how many URLs are in the queue, and more…

Learn more about the active sites
See what's happening

See what's happening

See when the URL collection, crawling, deleting, and recrawling events will be executed and for which site. You can also see the total number of posts in queue, crawled, recrawled, and deleted. While a post is being crawled or recrawled, it is also displayed in this section.

Learn more about seeing what's happening
Last crawled, recrawled, and deleted posts

Last crawled, recrawled, and deleted posts

See the latest posts that were crawled, recrawled, and deleted. You can also make the dashboard page refresh in specific time intervals to follow the events as they happen.

Learn more about recently crawled posts
See the Dashboard page in action
This video observes the dashboard page while the plugin crawls posts

Options box

Manipulates the value found by a CSS selector

Available for many settings

Available for many settings

The options box is displayed by clicking to the "settings" button, shown in the image with a blue border. The button is available for many settings that define a CSS selector.

Learn more about the options box

Proxies

Use proxies to access sites that block your server

Define proxies

Define proxies

You can define one or many proxy servers. While crawling, the plugin tries them one by one until it reaches the target website. The plugin uses these proxies every time it makes a request to the target site.

Learn more about using proxies

Manual crawling

Crawl posts manually with a powerful tool

Crawl multiple posts

Crawl multiple posts

Enter a post URL in a new line and hit the "Crawl now" button. The tool will crawl all the posts whose URLs are entered.

Retrieve post URLs from category pages

Retrieve post URLs from category pages

When you do not want to enter the post URLs manually, you can enter the URLs of category pages. The plugin extracts the post URLs from the category pages and crawls each found post URL.

Crawl in parallel

Crawl in parallel

The manual crawling tool is capable of crawling multiple posts in parallel. Just define how many posts you want to be crawled at the same time.

Add post URLs to the queue

Add post URLs to the queue

If you do not feel like crawling right now, you can choose to add the post URLs into the queue manually. Then, the plugin crawls them by using your scheduling options.

See the manual crawling tool in action
This video demonstrates the manual crawling tool by crawling a few posts manually

Notifications

Send yourself an email when something happens

Custom notifications

Custom notifications

By using the filters, you can define your custom conditions and send yourself a custom email when those conditions are met. You can include the URL of the post, HTTP status code of the response, name, ID and edit URL of the site, and current time in your email. You can also leave the email template empty to use the default template provided by the plugin.

Learn more about custom notifications

Interactive guides

See how to do things, step-by-step, interactively

Learn what to do

Learn what to do

The steps of the interactive guides show you exactly which setting to configure and how to configure it. The steps also check if the setting is correctly configured and show you what to do to fix the error when there is an error.

Start any guide whenever you want

Start any guide whenever you want

At the bottom right corner of the plugin's pages, there is a "Guides" button. When you click it, you see the list of interactive guides available in the plugin. From there, you can search for a guide and click its "play" button to start it. You can also start a guide from a specific step.

Watch the quick start guide
This video tutorial follows the interactive quick start guide and explains it along the way

and more…