Find unique elements
Unique CSS selectors select only one element. This is useful, for example, for finding titles and featured images.
Get content from almost any site, automatically!
WP Content Crawler is a WordPress plugin that helps you get content from other websites, into your website.
Now with ChatGPT!Save anything.
*Config Helper is displayed when creating a new site. Its purpose is to speed up the initial setup for a site. Although Config Helper works for many sites, not every site can be configured this easily, meaning that certain sites may require a manual and more elaborate setup. Even for those sites, you can try to create the basic configuration with Config Helper and then modify the site settings later. Additionally, it is possible to exit Config Helper any time or disable Config Helper entirely.
1
2
3
4
5
6
7
Find similar elements
The "similar" mode finds a CSS selector that selects the items similar to the clicked item. This is useful, for example, for finding post tags and product attributes.
Find containing elements
The "contains" mode finds a CSS selector that selects the element containing all the clicked elements. This is useful, for example, for finding post content and excerpt.
See the visual inspector in action
This video demonstrates the selection modes of the visual inspector.
Auto embed social media posts
Media from 70+ domains are automatically converted to embed short codes
HTML code of social media posts existing in the target post page is automatically converted to short codes. By this way, they are displayed on the front end of your website correctly. Additionally, all the iframe elements are converted to a short code. An iframe short code is displayed only if its source is trusted. If the source of an iframe is not trusted by default, you can manually add a trusted domain so that the iframe is displayed. With this method, you can display media from third-party sites securely. Websites whose media are embedded automatically include Instagram, Imgur, YouTube, Vimeo, TikTok, Kickstarter, Twitter, Pinterest, and so on.
Find and replace
Find and replace anything in the target page. You can also use regular expressions. You can use dedicated options to find and replace in the post title, content, tags, post meta, and others…
Learn more about finding and replacing
Manipulate HTML
You can remove and clone elements, remove and exchange element attributes, find and replace in element HTML and attributes, remove empty elements, remove links by domain…
Learn more about manipulating HTML
Advanced text manipulations
Make text upper case, lower case, title case, camel case, snake case, kebab case, and more… You can also limit the length by character count or word count.
Learn more about advanced text manipulations
Quick tests
Use the magnifier button to quickly test the configurations. You can test find-replace options, CSS selectors, formulas, filter commands, and more… You can also enable caching for the quick tests to make them even faster.
Test the settings altogether
The Site Tester page helps you check if your configurations work as expected. You just enter a category or post page URL and hit the test button. The plugin crawls the page and shows you all the details it finds in the page, along with some performance information and, if any, warnings.
Learn more about testing the settingsSee the Site Tester in action
This video demonstrates the features of the Site Tester page.
Transform, translate, and spin
Automatically transform, translate, and spin specific parts of the posts
Transform
You can transform the parts of the post by using paid OpenAI GPT models, like ChatGPT. Use the cutting-edge AI models to generate SEO-friendly summaries, insert interesting facts, rewrite posts, fix grammar errors, and do anything else that you can imagine.
Learn more about transformation
Translate
You can automatically translate the posts from one language to another by using one of the following paid services. You can also choose which parts of the post should be translated. Moreover, you can perform the translation multiple times and translate the element attributes.
Learn more about translating
Spin
Spinning is like paraphrasing, keeping the same meaning while using different words. You can automatically spin the posts by using one of the following paid services. You can also choose which parts of the post should be spun.
Learn more about spinning
The filter above assigns the status of the post as "draft" if the crawled post's title has less than 20 characters. Using many condition commands available in the plugin, you can create complex conditions. You can combine the conditions with "and" and "or" operators as well. When the condition is met, you can do multiple things, such as sending an email notification to yourself, adding the post to a specific category, finding and replacing things, assigning a specific featured image, calculating a value, and many more.
Learn how to use the filters
This video tutorial explains the filters in detail, with examples
Save WooCommerce products
Use the options dedicated to WooCommerce products to easily save the product details
Dedicated options
The WooCommerce section's design is similar to the design of the original WooCommerce options. Here, instead of entering the values, you define CSS selectors. You can create simple and external products. You can also mark the product as virtual and/or downloadable. You can retrieve the downloadable file from the target page as well.
Learn more about WooCommerce options
Crawling WooCommerce products
This video tutorial walks you through configuring the site settings to crawl products from eBay
Save custom post types
Save the crawled posts as custom post types instead of regular WordPress posts
Select any post type
Select the type of post you want to create, if you do not want regular WordPress posts. All the post types registered in your WordPress site are available. You can select a different post type for each site or set the post type for all the sites from the general settings.
Learn more about saving custom post types
Save custom fields (post meta)
Custom post types typically keep extra information in their custom fields. You can save the values of the custom fields in plain text format so that the custom post type can use them accordingly.
Learn more about saving custom fields
Save taxonomy values
Custom post types might also keep extra information in specific taxonomies, such as tags, company name, ISBN number, and so on. You can save the values of the taxonomies in plain text format so that the custom post type can use those values in the correct places.
Learn more about saving taxonomy values
Use custom post type categories
Custom post types might have their own category taxonomy. You can save the posts into those categories by defining the category taxonomy in the general settings. Then, you can select those categories in the Category URLs setting. The rest is handled by the plugin.
Learn more about using custom post type categories
Track active sites
See the active sites, when they were crawled the latest, how many posts were crawled, how many URLs are in the queue, and more…
Learn more about the active sites
See what's happening
See when the URL collection, crawling, deleting, and recrawling events will be executed and for which site. You can also see the total number of posts in queue, crawled, recrawled, and deleted. While a post is being crawled or recrawled, it is also displayed in this section.
Learn more about seeing what's happening
Last crawled, recrawled, and deleted posts
See the latest posts that were crawled, recrawled, and deleted. You can also make the dashboard page refresh in specific time intervals to follow the events as they happen.
Learn more about recently crawled postsSee the Dashboard page in action
This video observes the dashboard page while the plugin crawls posts
Available for many settings
The options box is displayed by clicking to the "settings" button, shown in the image with a blue border. The button is available for many settings that define a CSS selector.
Learn more about the options box
Automatically convert JSON to HTML
You can convert all the JSON data in the page automatically to HTML elements and use them in CSS selector options. This allows you to use the JSON data easily.
Learn more about converting JSON to HTML automatically
Convert JSON to HTML via your custom template
You can selectively retrieve parts from JSON data and put them into your custom HTML template. By this way, you can recreate an entire site defined as JSON. This makes it possible to retrieve data from pages that define their entire page as JSON and convert it to HTML later via JavaScript.
Learn more about converting JSON to HTML via a template
Make custom requests
You can make custom requests while crawling the target page. For example, you can make a request to an API, convert its JSON response to HTML, and use it in the post content. If the request URL does not exist in the target page, you can easily create an anchor element and inject it to the page. Watch this video for examples.
Learn more about making custom requests
Define proxies
You can define one or many proxy servers. While crawling, the plugin tries them one by one until it reaches the target website. The plugin uses these proxies every time it makes a request to the target site.
Learn more about using proxies
Randomize proxies
You can make the plugin randomly use a proxy from the list of proxies every time it makes a request to the target site.
Learn more about randomizing proxies
See the manual crawling tool in action
This video demonstrates the manual crawling tool by crawling a few posts manually
Empty value notifications
Define CSS selectors that must always find a value and then get an email when they do not have a value.
Learn more about empty value notifications
Custom notifications
By using the filters, you can define your custom conditions and send yourself a custom email when those conditions are met. You can include the URL of the post, HTTP status code of the response, name, ID and edit URL of the site, and current time in your email. You can also leave the email template empty to use the default template provided by the plugin.
Learn more about custom notifications
Start any guide whenever you want
At the bottom right corner of the plugin's pages, there is a "Guides" button. When you click it, you see the list of interactive guides available in the plugin. From there, you can search for a guide and click its "play" button to start it. You can also start a guide from a specific step.
Watch the quick start guide
This video tutorial follows the interactive quick start guide and explains it along the way
Cookies
Assign cookies to each request to crawl as a logged-in user
Headers
Add request headers to each request
Quick fixes
Do some standard operations by simply checking checkboxes
Prevent duplicate posts
Check for duplicate posts via URL, title, content
Customize the content
Add link to the source or anything to the post content
Define custom shortcodes
Assign an element to a shortcode and use it in the templates
Easily add category URLs
Fill the category URLs setting automatically
Password protection
Define password for the crawled posts
Custom general settings
Change the general settings per site
Set post status
Save posts as draft and check before publishing
Set post author
Assign the crawled posts to a custom user
Save all images
Save all images in the post content by checking a checkbox
Schedule posts
Change the publish date to a future date
Quick save
Save the changes in a few milliseconds
Fix character encoding
Convert the encoding to UTF-8 easily
Import/export settings
Easily clone or move the site settings to another website
and more…
Check out all the guides